Sparse Readout Prism: Explaining Logit-Lens Scores in Features Instead of Tokens
Matteo He, William F. Shen, Xinchi Qiu, Nicholas D. Lane
cs.CL · cs.AI · cs.LG
Quick verdict
SRP: an SAE over LM-head rows decomposes lens scores exactly; strong baselines and intervention checks, but cross-recipe feature stability is limited.
Summary
This method paper introduces Sparse Readout Prism (SRP), which trains a TopK sparse autoencoder on the rows of a language model's unembedding matrix (LM head) so that any lens score — a token logit or a logit difference — decomposes exactly into signed contributions from sparse readout features plus an explicit residual. Because the fit uses only readout weights, one dictionary per model serves every lens and layer. Across eight 0.8B–9B models, the sparse reconstruction preserves the held-out argmax on 0.89–0.90 of decoded states (0.75–0.76 for R1-distilled readouts) and covers 8.9–17.3 percentage points more of roughly 1,350 tested logit differences per model than the strongest of six row-geometry baselines; ablating feature directions shifts realized scores with r² of 0.83–0.93. The authors also demonstrate corpus conditionality: Jacobian lenses fitted on English versus Chinese corpora report different scripts on 39/80 prompts for identical hidden states, while the dominant readout feature agrees on 77/80. Limitations: ablation slopes exceed one (1.34–1.69) on Qwen readouts; the released dictionary shares only 36% of the stable feature core across recipes; softcapped Gemma readouts are excluded from the baseline comparison; coverage falls to 0.227 for near-tie contrasts.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Any lens score (token logit or logit difference) decomposes exactly into signed contributions from sparse readout features plus an explicit residual, using only the readout weights (§3).
- Replacing the readout with SRP's sparse approximation covers 8.9–17.3 percentage points more of the tested logit differences than the strongest of six row-geometry baselines (Table 1).
- Ablating a feature's decoder direction shifts the realized score in proportion to its SRP contribution, with r² of 0.83–0.93 against random controls at or below 0.06 (Appendix J).
- Jacobian lenses fitted on English versus Chinese corpora report different scripts on 39/80 prompts for identical hidden states, while the dominant readout feature agrees on 77/80 (§5).
- The dominant readout feature remains stable across fitting languages, two lens constructions, a shared-script pair, and corpus sizes, even as reported tokens vary (§5.3).
Strengths
Exact-by-construction decomposition with always-on fidelity diagnostics
The score split (§3.2, Appendix A) sums to the exact logit by identity, and every display reports residual, sign agreement, and relative error ρ; operating-regime gating (Appendix G.6) routes failing cases to weaker readings instead of hiding them.
Capacity-matched baselines make the comparison meaningful
The k-means centroid dictionary at matched D=65,536 and k=256 (Table 21) grants alternatives SRP's exact capacity; SRP still leads on all six softcap-free readouts (e.g., Qwen3.5-9B 0.812 vs 0.639 coverage), with full error tails in Table 22.
Falsifiable intervention validation
Well-controlled corpus conditionality study
Unusual transparency about failures
Weaknesses
Feature identity is unstable across recipes
Intervention claim is proportional but not calibrated
Headline coverage includes an unreliable near-tie regime
Corpus conditionality is tested at small scale on one model
Curated comparison set: softcapped readouts excluded
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SRP targets the readout side of lens interpretability. Lens readings (logit lens, tuned lens, Jacobian lens) are consumed as token rankings, but token identity conflates senses within one unembedding row and splits shared structure across rows (Figure 1). The paper makes four claims: (1) an exact, corpus-free decomposition of any selected readout score into signed feature contributions plus residual (§3.2); (2) reconstruction fidelity exceeding six row-geometry baselines by 8.9–17.3 percentage points of coverage (Table 1, six softcap-free readouts); (3) predictive validity of contributions under ablation (Appendix J); and (4) corpus conditionality of fitted lenses with feature-level stability (§5). The interpretation boundary is stated honestly: claims are scoped to selected readout scores, not circuits or generation (Limitations).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Sparse Readout Prism (SRP)?
A method that trains a sparse autoencoder on the rows of a language model's unembedding matrix (LM head), so any lens score — a token logit or a logit difference — can be rewritten exactly as signed contributions from sparse readout features plus a small residual. It changes what a logit-lens reading reports, from token identities to features, without changing the lens itself.
What is corpus conditionality?
The paper's term for the finding that a fitted lens can report different tokens for the same hidden state depending on the corpus it was fitted on. Jacobian lenses fitted on English versus Chinese C4 prompts gave different-script answers on 39 of 80 prompts, yet the dominant SRP readout feature was the same on 77 of 80.
Does SRP require training data or a corpus?
No corpus enters the fit: SRP uses only the readout weights, which is why the authors propose it as a corpus-independent control for lens analyses. Training one dictionary took about 6–12 GPU-hours on an NVIDIA A40; code (MIT) and per-model dictionaries are publicly released.
How faithful is the sparse readout replacement?
On six softcap-free models the reconstructed head preserves the original argmax on 0.89–0.90 of held-out states (0.75–0.76 for R1-distilled readouts), sign agreement on selected scores is at least 0.91, and reliability drops mainly for near-tied contrasts with margins below 0.5 logits.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with several marked '[...sampling gap...]' sections; §4.3–4.4 (the grouping-stability results behind the 13–15× seed-overlap claim) and the full contents of Table 2 were not visible, so those results are assessed from visible summaries in §7 and Appendix I rather than their primary tables.
- Several appendix tables (e.g., Tables 7–8, 18, 26, 36) appear only by reference in the excerpt; per-family cross-lens numbers were verified from Table 32 and surrounding visible text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.