Where Decoder Cosine Similarity Fails for SAE Feature Flow Discovery
Hendrik Droste, Christian Medeiros Adriano, Kathrin Korte, Holger Giese
cs.LG
Quick verdict
Claims pairwise SAE decoder cosine misses most ablation-validated feature transitions (88% Pythia, 54% Gemma subset); solid causal method, thin generality.
Summary
The paper asks whether strong cross-layer feature transitions in transformers can be recovered from pairwise SAE decoder cosine similarity alone, or whether they require the joint presence of a residual-state feature and an MLP-update feature. The authors build a 'transition atlas' of triples s_k + u_j → t_ℓ, scoring candidates with a normalized second-order interaction statistic and validating them causally by ablating the decoded update feature from the MLP output and requiring a target-feature change of at least two standard deviations. In a full-atlas Pythia-160M L7→L8 run (20M scoring tokens, custom TopK SAEs, 262,735 filtered triples), 38,125 triples (14.5%) pass validation, yet 88.0% have both state-target and update-target cosine below 0.7 (62.1% below 0.4), answering the research question negatively. A complementary state-to-update ablation shows 91.9% of the top 4,096 (s,u) pairs produce strong effects (median Az 40.6), motivating the triple-level framing. A preliminary Gemma-3-4B L21→L22 check with pretrained Gemma Scope SAEs validates only the top 30,000 ranked candidates: 53.6% of its 12,139 strong-effect triples fall below 0.7 on both cosines — directionally consistent but weaker, and not a full-atlas recall estimate. Key limitations: one fully validated layer pair, subset-based Gemma validation, no random-ablation controls, no confidence intervals, and no stated code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- In the Pythia-160M L7→L8 full atlas, 88.0% of 38,125 ablation-validated strong transitions (|Az| ≥ 2) have both state-target and update-target decoder cosine below 0.7, so pairwise decoder cosine cannot recover most strong causal feature transitions.
- The recall failure is not a high-threshold artifact: 62.1% of strong-effect triples still have both pairwise cosines below τ = 0.4.
- State features causally condition MLP update features: 91.9% of the top 4,096 ranked (s,u) pairs show |Az| ≥ 2 with median effect 40.6 on 500k tokens.
- A preliminary Gemma-3-4B L21→L22 run validating only the top 30,000 ranked candidates is directionally consistent but weaker: 53.6% of 12,139 strong-effect triples have both cosines below 0.7, and update-target cosine recovers 68.2% of strong effects.
- Feature-flow atlases scored by second-order interaction and validated by ablation are therefore needed as diagnostics beyond pairwise decoder geometry.
Strengths
Causal yardstick instead of geometry-only claims
Strong-effect labels come from ablating the decoded update feature out of the MLP output and re-encoding (§2.1, A.1.5), with effects normalized by the target feature's own std (Az), explicitly motivated by the cross-feature scale problem (Sun et al., 2024). The core conclusion therefore rests on interventions, not correlational geometry.
Full-atlas Pythia run with complete accounting
Table 1 tracks 199,004,232 unfiltered → 262,735 filtered → all 262,735 validated triples, making the headline 88.0% (33,550/38,125 both cosines < 0.7) a population-level statement for the L7→L8 pair rather than a sample estimate.
Honest hedging of the cross-model evidence
Threshold-robustness check pre-empts an obvious objection
Reproducibility scaffolding in the appendix
Weaknesses
Generality rests on one full layer pair plus a non-random subset check
Validation criterion is uncalibrated
Key pipeline constants never given numerically
Descriptive statistics without uncertainty
Scope narrower than the framing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.12591) addresses a narrow, well-posed question in mechanistic interpretability: can strong cross-layer feature transitions be recovered from pairwise SAE decoder cosine similarity alone, or do they require the joint presence of a residual-state feature and an MLP-update feature (§1, §2)? The deliverable is a "transition atlas" of triples s_k + u_j → t_ℓ built with a three-stage pipeline (§2.1): sparse event extraction (top 64 features/token), candidate construction with a normalized second-order interaction score, and ablation-based causal validation requiring |Az| ≥ 2 target-feature standard deviations. Central claims: (1) in a full-atlas Pythia-160M L7→L8 run, 88.0% of 38,125 validated strong-effect triples have both pairwise cosines below 0.7 (62.1% below 0.4); (2) state features causally condition update features (91.9% of the top 4,096 pairs, median Az = 40.6); (3) a preliminary Gemma-3-4B L21→L22 check on the top 30,000 ranked candidates is directionally consistent but weaker (53.6% of 12,139 strong effects below 0.7). The title's framing ("where decoder cosine similarity fails") is accurate for the tested configurations but broader than the evidence base — a point the authors themselves concede in §4.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'decoder cosine similarity' fail at in this paper?
It fails to recover most causally validated cross-layer feature transitions. In the authors' Pythia-160M L7→L8 experiment, 88.0% of transitions confirmed by ablating an MLP-update feature (target change of at least 2 standard deviations) had both state-target and update-target decoder cosine below 0.7 — meaning a cosine-threshold search would miss them, and 62.1% were still missed at a 0.4 threshold.
What is a 'transition atlas' of SAE features?
A ranked list of triples s+u→t, where a residual-state feature and an MLP-update feature jointly predict activation of a target residual feature. Candidates are scored by a second-order interaction statistic over token activations, then confirmed causally by ablating the decoded update feature from the MLP output and checking that the target feature changes by at least two of its own standard deviations.
Which models were used, and is the finding replicated across models?
The full result is on Pythia-160M layer 7→8 with custom TopK SAEs. A preliminary check on Gemma-3-4B layer 21→22 with pretrained Gemma Scope SAEs validated only the top 30,000 ranked candidates; it is directionally consistent (53.6% of 12,139 strong-effect triples have both cosines below 0.7) but weaker, since update-target cosine recovers many strong Gemma effects, and it is not a full-atlas recall measurement.
Is code available for this paper?
No code or data release is stated in the paper text. The appendix specifies SAE training hyperparameters and the pruning formula, but several filtering constants (the per-token pruning count and activation/co-occurrence thresholds) are not given numerically, so exact replication currently requires re-deriving those choices.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2's plotted distributions are available only through the caption and prose description; the extracted text does not include the figure images, so characterizations of the scatter's shape rest on the authors' own description.
- The metadata abstract is cut mid-sentence ('and the run'), but the full-text abstract section is complete; no part of the provided paper text itself appears truncated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.