Through the Looking Glass: Directly Reading and Writing Transformers
cs.CL · cs.LG
Quick verdict
Signed attribution cuts per-token circuits to dozens; naming and editing follow without training. Strong controls; depth centers on one self-trained baseline.
Summary
The paper asks how many transformer components decide a single token and argues that the standard absolute-contribution count is inflated because signed contributions cancel: across eighteen models, mass opposing the predicted token is a median seven times the mass supporting it. Dividing by the net shrinks counts from thousands to dozens — on the authors' self-trained GPT-2-small baseline, 53 components carry 90% of a prediction (n90), 13 are necessary, and a repair-and-prune search finds 8 sufficient. On twelve external models (124M–7B parameters, seven architecture families), sufficient sets run 2–16 components and the full dependency closure of a prediction is 1–3% of the model, flat in size. Components are named without training anything: write sides from the predictions they drive (covering close to half of all components), read sides from weights decoded against a measured layer-native token table (58.9% ov@8 versus 7% via vocabulary projection, falling below chance at K=256). Names are actionable: installing an association into one spare unit moves a token from rank 578 to rank 1 at ~0.25% held-out loss, reaching the top ten roughly forty times more cheaply than a rank-one editor. Holding the model fixed and swapping GELU for a monotone activation raises read agreement to 96–99%, flat in K. Limitations: deep analysis rests on one self-trained baseline (3 of 8 seeds diverged), sufficient sets are heuristic upper bounds, and parallel-attention architectures (Pythia) are excluded.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Counted by net signed contribution rather than absolute magnitude, a single token prediction rests on dozens of components (median 53 at the 90% bar on the baseline), not the thousands the standard convention reports.
- On the baseline, 13 components are necessary and 8 are sufficient for one prediction; across twelve external models (124M–7B) sufficient sets run 2–16 and a prediction's dependency closure is 1–3% of the model, flat in size.
- Components can be named on both sides with nothing trained: writes from the predictions they drive (close to half the model, Table 13), reads from weights decoded against a measured layer-native token table (58.9% ov@8 vs 7% through the vocabulary).
- An association the model does not hold installs into one spare unit via weight-read key and value, moving the target from rank 578 to rank 1 for ~0.25% held-out loss — about a fortieth of a rank-one editor's cost to reach the top ten.
- GELU's non-monotonicity (decreasing over ~60% of pre-activation mass) explains why weight-based reads decay below chance, and order-preserving activations restore flat, near-ceiling agreement (96–99%).
Strengths
The signed-vs-absolute accounting is quantitatively decisive, not anecdotal
Opposing mass is a median 7x the net across eighteen models; Table 25 shows the absolute convention would inflate n90 by 98x (ReLU arm) to 632x (unshaped sigmoid), and Table 6 shows net-ranking wins specificity at every k (5.5 vs 3.2 absolute-ranked at k=64).
Unusual methodological candor with failure modes documented in-line
Table 1 shows the three excluded seeds (ppl 21.6–39.5) because collapsed models read as more legible (their ov@8 is 40.7–49.2 vs 33.6–38.4 converged); Table 14 pins each measurement to a condition defusing a specific flattering artifact; Section 2.4 replaces raw logit drops with a margin to defeat final-norm shrinkage (45% sign disagreement on some external models).
The layer-native token table is a measured, non-fitted alternative to trained explainers
Editing claims ship with their own controls
Breadth with a principled scope
Weaknesses
All depth analysis rests on one in-house model with a troubling seed record
Sufficient sets are heuristic upper bounds that consume the answer
The GELU mass-fraction diagnosis may be outlier-driven
The rank-one comparison is single-layer and endpoint-mismatched
The carriage claim is a strong architectural generalization from ablation economics
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper builds the Looking Glass, a training-free instrument for reading and editing transformers, and makes five load-bearing claims. First, absolute-value attribution overcounts: contributions to a logit are signed, the mass opposing the predicted token is a median 7x the mass supporting it across eighteen models, and net counting returns dozens of components where absolute counting returns thousands (inflation 98x–632x across arms, Table 25). Second, on the authors' GPT-2-small baseline, 53 components carry 90% of a prediction (n90, median over 115 predictions), 13 are necessary by prefix scan, and 8 suffice by a repair-and-prune search (Table 7; 0.017% of 46,080 components). Third, on twelve external models (124M–7B, seven families), sufficient sets run 2–16 and a prediction's dependency closure is 1–3% of the model, flat in scale. Fourth, components are nameable without any fitted explainer: writes from driven predictions (48.2% named at the n90 operating point, Table 13), reads from weights decoded against the measured layer-native token table of Eq. 7 (58.9% ov@8 vs 7% via vocabulary projection; below chance at K=256, Figure 7). Fifth, names are actionable: a one-unit install moves a token from rank 578 to 1 for ~0.25% held-out loss and reaches top-ten about forty times more cheaply than a rank-one editor (Section 4.9).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main idea of 'Through the Looking Glass'?
It argues that a transformer's single-token prediction is carried by only a few dozen components once contributions are counted with their sign (net) instead of by absolute magnitude, and that the model's own weights and activations — with nothing trained or fitted — suffice to name what components read and write and to edit them.
How much smaller are the circuits than prior estimates?
On the authors' GPT-2-small baseline, 53 components carry 90% of a prediction's net contribution, 13 are necessary, and 8 suffice to reproduce the token; the same predictions scored by the usual absolute-value convention implicate roughly 24,000 components.
Does the method need a trained explainer like a sparse autoencoder?
No. The paper deliberately avoids fitted explainers: it decodes component weights against a measured layer-native token table and names components by the predictions they drive, so every quantity is read from the model's parameters and activations. It reproduces SAE-style results in an appendix for comparison.
Can the method be applied to any LLM?
In the paper it applies to models with the sequential attention-then-FFN arrangement, tested on twelve public models from 124M to 7B parameters across seven architecture families. Parallel-layout models such as Pythia/GPT-NeoX are explicitly excluded and left to future work, and very large models strain the circuit search.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with [...sampling gap...] markers; Sections 5.3–5.4, most of Section 6, Sections 7–9 and the appendices are only partially visible, so assessments of those parts rest on the abstract and surviving fragments.
- No code or data availability statement appears anywhere in the provided excerpt; one may exist in the skipped portions.
- Figure content extracts as garbled inline text (e.g., the Figure 4 trace tree), though captions and the numbers embedded in them remain readable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.