Signatures of Steerability in Activation Space of Language Models
Prajjwal Bhattarai, Tuka Alhanai
cs.LG · cs.AI
Quick verdict
Shows separability metrics—above all SNR—strongly predict CAA steering success across 6 models and 38 datasets; correlational, multiple-choice-only, so no causal or open-ended claims.
Summary
The paper asks whether the success of a steering vector can be predicted before applying it. The authors compute four separability metrics—signal-to-noise ratio (SNR), mean μ-alignment, manifold capacity, and intrinsic dimension—from the contrastive activations used to train CAA steering vectors, and correlate them with steerability (the slope between steering multiplier and mean positive-minus-negative logit propensity) for six instruction-tuned models from the Qwen 3, Gemma 3, and Llama families over 38 persona, truthfulness, and safety datasets. SNR is the most consistent predictor, with Spearman correlations of 0.62-0.83 per model family, and it retains partial correlations of 0.72-0.74 after controlling for dataset and layer effects. A synthetic superposition experiment probes the mechanism: SNR and mean alignment correlate strongly with feature recovery, the cosine between the computed vector and the true feature direction (ρ=0.91 and 0.90), but weakly with sparsity or dictionary coherence. The key limitation is that the whole analysis is correlational, restricted to one steering method, multiple-choice datasets, and logit-based propensities, with no fluency measurement and no causal test.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Simple separability statistics computed from contrastive training activations — especially SNR — strongly predict downstream CAA steerability across six instruction-tuned models and 38 datasets (up to r=0.82 per model; Table 1).
- The SNR-steerability association survives controls for dataset and layer effects (partial correlations 0.74/0.56 and 0.74/0.72; Table 2), supporting the claim that steerability is a function of the source data's geometry rather than the model.
- In a synthetic superposition setting, separation metrics primarily track feature recovery (SNR vs. π: ρ=0.91, Table 3) rather than superposition sparsity (k: −0.33) or dictionary coherence (ε: −0.01, n.s.).
- Intrinsic dimension of difference vectors is not a reliable predictor: its correlation is weak, flips sign across dataset families (Tables 7–9), and collapses when controlling for layers (Table 5).
- All evidence is correlative, not causal (Sec. 6.2): the authors explicitly leave open whether separation contributes causally to steerability or both are driven by a third variable.
Strengths
Breadth and controlled design
Six instruction-tuned models across Qwen 3, Gemma 3, and Llama at two scales, with 26 persona + 5 truthfulness + 7 safety datasets (Sec. 3.1), a 50/50 train/test split isolating metric computation from steerability measurement, and partial correlations after residualizing on dataset or layer indicators (Tables 2, 5, 6).
Consistent, robust headline result
SNR is the top predictor in every model-family cell (Pearson 0.69–0.82, Table 1), remains best after controls (0.74/0.56 controlling dataset, 0.74/0.72 controlling layers, Table 2), and its dominance holds within the persona, safety, and truthfulness families (Tables 7–9).
Mechanism credit assignment, not just prediction
Honest uncertainty and negative results
Useful practitioner deliverable
Weaknesses
Correlational by design; no causal manipulation
Partial-correlation machinery is thin
Single steering site and proxy evaluation
Synthetic setting is a stylized linear world
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses a practical question left open by Tan et al. (2024) and Braun et al. (2025): can the success of a steering vector be predicted from the activations it is built on, before any intervention? The authors claim (Abstract, Sec. 4) that simple separability statistics of contrastive activations — above all a signal-to-noise ratio (SNR) — correlate strongly with downstream CAA steerability across models, datasets, and layers, and that a synthetic superposition experiment (Sec. 4.2) explains *why*: the metrics track recovery of the underlying feature direction rather than global superposition geometry. The contribution is a diagnostic, not a new steering method, and the paper is commendably explicit that its evidence is correlative (Sec. 6.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper 'Signatures of Steerability in Activation Space of Language Models' show?
It shows that simple statistics computed from the activations used to train a steering vector — most consistently a signal-to-noise ratio (SNR) — strongly predict how well that vector will steer a language model afterward. Across six instruction-tuned models (Qwen 3, Gemma 3, Llama) and 38 persona, truthfulness, and safety datasets, SNR correlates with steerability at Spearman 0.62–0.83, and remains around 0.72–0.74 after controlling for dataset and layer effects. A synthetic experiment suggests this works because separation tracks whether the computed vector aligns with the true underlying feature direction. The evidence is correlational, so it does not prove separation causes steerability.
How can I predict whether an activation steering vector will work before applying it?
According to this paper, compute a separation statistic on your contrastive training activations before steering. The proposed signal-to-noise ratio is the squared norm of the mean difference vector divided by the sum of the two class variances' traces. Higher values predicted stronger steering response (the slope between steering strength and the logit-difference propensity) across Qwen 3, Gemma 3, and Llama models and 38 multiple-choice datasets. Caveats: results were established only for CAA-style vectors, multiple-choice evaluations, and a correlational design — it remains untested whether high-separation steering preserves output fluency in free-form generation.
What is CAA (Contrastive Activation Addition) and steerability?
CAA (Rimsky et al., 2024) builds a steering vector as the mean difference between activations from contrastive prompt pairs (desired vs. undesired completion), then adds a scaled copy of this vector to the model's activations at inference time. This paper measures steerability following Tan et al. (2024): the model is steered with increasing multipliers λ, the average propensity (logit difference between the positive and negative answer choices) is recorded, and steerability is the slope of that line — a larger slope means steering more reliably moves the model toward the target behavior.
Does the paper prove that separation causes better steering?
No. The authors state explicitly (Sec. 6.2) that all analyses are correlative. They cannot rule out that an unknown third variable drives both geometric separation and steerability, and they did not run any intervention that manipulates separation directly to observe the effect on steering. The synthetic superposition experiment supports a plausible mechanism — separation metrics track alignment between the computed vector and the true feature direction — but that setting is an exactly linear generator, not a real language model.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Only the paper's text was analyzed; Figures 1's scatter plots cannot be inspected, so the visual distribution behind the r=0.76 aggregate is unverifiable.
- Several layer/multiplier configuration details (exact layer set, Λ grid, per-cell observation counts) do not appear in the extracted text and may live in unlabeled appendix portions.
- The per-model n (number of dataset-layer pairs) is never stated, which limits assessment of correlation precision.
- Some cited works are dated 2025-2026 and could not be verified offline.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.