Decodability is Not Causality: Dissociating Probe Readouts from Behavioral Drivers via SAE Decomposition
Devesh Tiwari, Camille Davis, Shivank Sinha, Talia Weaver, Aditya Shah, Maheep Chaudhary
cs.AI
Quick verdict
A workshop-scale but methodologically careful SAE-intervention study showing that probe weight geometry does not identify the Gemma-2-9B features that causally control a True/False truth probe's behavior — convincing within its single model/layer/task setting, but generalization is untested.
Summary
The paper asks whether SAE features geometrically aligned with a linear truth probe are the features the model causally uses. Using the TTPD probe (Bürger et al., 2024) on Gemma-2-9B-Instruct under instructed deception (Long et al., 2025), the authors decompose layer-20 residual activations with a GemmaScope SAE (16,384 features), rank features by probe alignment |w·d| and by gradient sensitivity of the True/False logit margin, then ablate matched feature sets while tracking behavior flips, probe readout shift, and output coherence. Shared (probe∩gradient) features flip up to 27% of outputs versus 6% probe-only and 1% random at perfect coherence; rankings overlap only ~12% (Spearman ρ=0.10); an activation-aware learned mask flips 17.6% versus 6.1% for the geometric top-16; ablating the full TTPD direction flips 18.1%, of which the 16 shared features recover two-thirds. Results hold over five seeds and a held-out split. Limitations: one model, layer, SAE, and binary task; flips are directional (mostly True→False, biasing toward False rather than restoring truth); gradient attribution alone is a weak behavioral predictor, so the 'model ranking' entering the shared set is itself noisy.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Sensitivity of the model's True/False behavior to SAE features is not recoverable from probe weight geometry alone: the top probe-aligned and top gradient-sensitive features overlap only ~12% (Spearman ρ=0.10).
- Ablating shared probe–model features flips model outputs up to 27% at perfect coherence, versus 6% for probe-only and 1% for random features of equal size, and this ordering holds across five seeds and a held-out split.
- Probe-only features mainly perturb the probe's own readout (relative shift 0.24 vs 0.05) without moving behavior, so geometrically probe-aligned features act as diagnostic readouts rather than behavioral drivers.
- An activation-aware selection (learned sparse mask) flips behavior 17.6% versus 6.1% for the geometric top-16, showing the failure is specific to activation-free geometric alignment.
- The full TTPD direction is itself causal (18.1% flips), and the 16 shared features localize two-thirds of that effect.
Strengths
Rigorous control structure in the ablation design
Section 3.6/5.2 compares size-matched shared, probe-only, and random sets (13–27% vs 6% vs 1% flips) and separates behavioral effect from probe-readout shift, so the dissociation is not confounded by set size or by probe perturbation per se.
Coherence and perturbation-magnitude controls guard against trivial explanations
Section 3.7 reports coherence 1.00 throughout, small residual-norm removal (5.7% for shared vs 13.9% for geometric top-16 in Sec. 5.3), and low KL (0.11 nats), plus SAE reconstruction quality (cosine 0.89); Section 5.3 uses the norm comparison to rule out 'bigger perturbation = bigger effect'.
Robustness and generality checks beyond a single run
Constructive refinement, not just a negative result
The SAE-substrate verification is a transferable diagnostic
Weaknesses
Extreme narrowness of scope
Fragile 'model ranking' component of the headline shared set
Thin behavioral readout and modest effect sizes
Underpowered / absent formal statistics for key contrasts
Internal numbering inconsistencies and a possible Table 1 typo
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'decodability is not causality' mean in this paper?
A linear probe can accurately decode information (here, truthfulness) from a language model's internal activations, but that does not prove the features the probe weights are the ones the model actually uses to produce its behavior. The paper tests this by ranking SAE features two ways — by geometric alignment with the probe's weight vector and by gradient sensitivity of the model's output — and then ablating each set to see what actually changes the model's True/False answers. The two rankings barely coincide, and the features the model is sensitive to are not primarily the ones the probe points at.
Did the paper find that the truth probe is useless?
No. The probe direction itself is causally load-bearing: projecting it out of the residual stream flips 18.1% of the model's True/False answers. The finer finding is about attribution: the SAE features most aligned with the probe's weight vector are mainly diagnostic readouts, while 16 features that are both probe-aligned and gradient-sensitive recover two-thirds of the probe direction's behavioral effect. The failure is specific to activation-free geometric ranking; combining probe information with activation statistics recovers much more causally effective features.
Is this result general, or specific to one model and task?
It is demonstrated in a single setting: Gemma-2-9B-Instruct, layer 20, one GemmaScope SAE, and a True/False factual-verification task under instructed deception. Effects appear across twelve dataset categories, five seeds, and a held-out split, but an auxiliary test on a Qwen3-8B reward-hacking probe failed the paper's SAE-fidelity precondition, so no second positive case exists yet. The authors themselves flag generalization across models, layers, SAEs, and behaviors as open.
Did ablating the shared features make the model more honest?
No — and the paper is explicit about this. The flips are overwhelmingly True→False (86 vs 4 out of 800 statements), meaning ablation biases the decision variable toward False regardless of factual correctness, and accuracy under the deceptive prompt drops from 0.403 to 0.330. The result demonstrates causal control over the output, not honesty restoration.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-9 are described only via captions in the extracted text; quantitative claims drawn from them (e.g., the N=96 peak of 0.31) could not be visually verified.
- Some numbers are internally inconsistent across sections and tables (0.130 vs 0.121; 0.061 vs 0.068; Table 1 chance 2.44 vs expected 0.61), suggesting extraction or typesetting issues; the review treats the tabled five-seed values as authoritative.
- The text mentions code 'at this URL' but the URL is not visible in the extraction, so reproducibility is assessed on stated component availability plus the code claim.
- Author affiliations are mostly high schools; this does not affect the technical assessment but contextualizes the workshop-scale scope of the study.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.