Disentangling Steering Vectors
Takeru Hiramatsu, Kyohei Atarashi, Koh Takeuchi, Hisashi Kashima
cs.LG
Quick verdict
Proposes SAE dissection of DiffMean steering vectors into interpretable concepts; evidence is broad and honest but judge-dependent and thinly baselined.
Summary
The paper introduces Steering Vector Dissection, a pipeline that treats conventional DiffMean steering vectors as composite objects and decomposes them into interpretable components. The method pairs positive and negative token activations via batch-wise cosine nearest-neighbor matching, computes instance-level difference vectors, and trains a small BatchTopK SAE (16 latents, bias-free decoder, no centering) directly on these differences; decoder columns become candidate basis vectors, which an LLM-judge pipeline interprets and filters. Experiments on PubMedQA (vs Alpaca) and PKU-SafeRLHF (unsafe vs safe) with Qwen2.5-7B and Llama3.1-8B at two depths show basis vectors achieve higher concept-extraction consistency than DiffMean in most settings (exception: Qwen L13 on PKU-SafeRLHF, 0.438 avg vs DiffMean 0.504), induce more consistent embedding shifts, and produce mutually distinguishable effects. As an application, ablating identified rejection directions from the unsafe DiffMean vector raises StrongREJECT scores markedly in earlier layers (Qwen L13: 0.057 to 0.352), though gains are small or null in two of four pairs. An SVD comparison, latent-dimension sweep, seed replication, and random-selection control support the design. Main limitations: all evaluation stages rely on LLM judges without human validation or inter-judge agreement data, the key Section 6 baseline (Arditi et al.'s refusal direction) is not compared, statistical testing is thin, and code release is not stated.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Conventional DiffMean steering vectors are composite objects, and an SAE trained on paired instance-level activation differences recovers decoder basis vectors that isolate individual, semantically consistent concepts (Sections 3, 5).
- Filtered basis vectors achieve higher concept-extraction consistency than DiffMean in most of the eight model-layer-dataset settings and induce mutually distinguishable steering effects (Table 3, Figure 4).
- Unsafe DiffMean vectors contain rejection-related components, and ablating these raises StrongREJECT substantially in earlier layers (Qwen L13 from 0.057 to 0.352) with small or null effects in two of four pairs (Table 6).
- Relative to SVD, the SAE yields more numerous and finer-grained concepts as latent dimensionality grows (surviving vectors 7 to 33 vs 7 to 11 at 16 to 128 latents) and larger effects in the rejection-ablation application (Section 7, Appendix C).
Strengths
Well-motivated problem framing and simple, fully specified pipeline
Sections 1-3 ground the entanglement problem in documented side effects (Zou et al. 2023 happiness vector; Durmus et al. 2024 bias co-amplification); the pipeline (batch cosine nearest-neighbor pairing, instance-level differences, 16-latent BatchTopK SAE with bias-free decoder and no centering) and steering normalization (Eq. 11-12, Table 11 coefficients 0.6-1.2) are specified precisely enough to replicate.
Unusually thorough controls and ablations for an interpretability paper
SVD top-16 replacement (Section 7, Tables 7-10), latent-dimension sweep 16 to 128 showing SAE surviving vectors grow 7 to 33 while SVD stalls at 7-11 (Appendix C, Table 16), second-seed replication of all headline results (Appendix B, Tables 12-15), random-vector baseline isolating the filtering contribution (Appendix D, Table 19: SAE 36 vs random 13 surviving vectors), and DiffMean reconstruction check at cosine 0.96 with all surviving dimensions active (Appendix E).
Honest, carefully scoped epistemic framing
Downstream claim validated with a quantitative benchmark, CIs, and seed replication
Weaknesses
Every evaluation stage runs through LLM judges, with circularity between filtering and the headline metric
Missing the obvious Section 6 baseline
Thin statistical reporting with a headline result driven by one cell
Bounded generality and no code release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses a real weakness in activation steering: DiffMean-style vectors are known to bundle multiple concepts (motivated by Zou et al. 2023's happiness vector increasing harmful compliance, and Durmus et al. 2024's feature co-amplifying age bias). The claim structure is carefully scoped — this is explicitly an existence proof (Section 5.2), not a claim of canonical or complete decomposition (Section 8). Two central claims are broadly supported: (1) filtered SAE basis vectors are more concept-consistent than DiffMean in most of the eight model-layer-dataset cells (Table 3), and (2) removing judge-identified rejection directions from an unsafe DiffMean vector raises StrongREJECT in earlier layers (Table 6). The mechanism narrative — that DiffMean contains rejection components which cancel the intended unsafe effect — is an inference from Table 6's pattern (DiffMean itself fails to raise compliance in 3 of 4 pairs); it is plausible but not directly established.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Steering Vector Dissection?
It is a pipeline that decomposes a composite DiffMean steering vector into interpretable pieces: it pairs positive and negative activations of an LLM, trains a small sparse autoencoder (16 latents, bias-free decoder) on the activation differences, and then uses LLM judges to interpret and filter the autoencoder's decoder columns into concept-specific, usable steering directions.
Does removing rejection directions make LLMs easier to jailbreak?
In the paper, ablating identified rejection components from an unsafe DiffMean vector sharply raises StrongREJECT jailbreak scores in earlier layers (e.g., Qwen2.5-7B layer 13 from 0.057 to 0.352), but effects are small or null in two of four model-layer pairs. The authors note this requires white-box access to activations plus a safety-labeled dataset, and discuss the dual-use risk explicitly.
How is this different from training a normal SAE on activations?
Instead of raw activations, the SAE is trained on differences between paired positive and negative activations — the same operation that produces DiffMean steering vectors — using a small latent space (16), a bias-free decoder, and no input centering, so decoder columns directly span the steering-vector space rather than general activation space.
What datasets and models were used in the paper?
PubMedQA (with Alpaca as the generic negative dataset) and PKU-SafeRLHF (unsafe severity-3 vs safe severity-0 responses), 8,192 prompt-response pairs each, evaluated on Qwen2.5-7B-Instruct and Llama3.1-8B-Instruct at two intermediate depths (layers 13/20 for Qwen, 15/23 for Llama), with StrongREJECT used for the jailbreak-relevant application.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Only textual content was extracted: the plot-level values of Figures 1-3 and 5 (e.g., exact embedding-shift similarities per strength in Figure 3) are not in the text, so those results were assessed from captions, stated trends, and the numeric matrices of Figure 4.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.