The Misery of Mechanistic Interpretability: A Formal Perspective
Tobias Ladner, Matthias Althoff
cs.LG · cs.AI
Quick verdict
First formal per-layer faithfulness certificates for LLM SAEs/transcoders; sound and novel, but local guarantees only and certified overlap stays low.
Summary
The paper argues that interpretations read off interpretable replacement networks (IRNs) — sparse autoencoders (SAEs) and transcoders — are unstable under small, semantics-preserving paraphrases, and proposes the first formal verification framework for IRN faithfulness in language models. Empirically (Sec. 3, Tab. 1), synonym- and LLM-based paraphrase attacks substantially reduce top-20 feature overlap between clean and perturbed activations across GPT-2, Gemma 2 2B, Gemma 3 1B, Llama 3.2 1B, and R1-Distill-Qwen 1.5B (e.g., the layer-averaged Jaccard of the GPT-2 SAE falls from 0.37 at MINOR to 0.13 at MAJOR level). The formal framework (Sec. 4) encodes per-token paraphrase hulls as zonotopes, builds difference networks (MLP minus transcoder; identity minus SAE), and certifies an ℓ∞ bound δ on the faithfulness gap via CORA reachability analysis; verified gaps grow with layer depth. Verification-aware training (SET) plus LoRA (Sec. 5) tightens the certified δ by roughly 90% at GPT-2 layer 6 while increasing sparsity (13.9 active features, −81%), yet only 2–3 of the top-20 features are provably retained, which the authors attribute to the base model's own fragility (21% next-token flips under MINOR paraphrase). Main limitations: guarantees are per-layer and local, with whole-model compositional verification left as a vision (Sec. 6); the 'sufficiently small' δ threshold is undefined; targets are deliberately maximally polysemous; and statistical reporting is descriptive only.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Minor semantics-preserving paraphrases flip the dominant IRN features across five open-weight model families, with worst-layer top-20 Jaccard overlap reduced by 73% at MINOR and 95% at MAJOR level (Tab. 1).
- Reachability analysis (CORA, zonotopes) certifies sound per-layer upper bounds on the ℓ∞ faithfulness gap between an LLM block and its IRN, and the verified gap grows with layer depth (Fig. 4).
- Verification-aware training (SET) plus LoRA tightens the certified δ by ≈90% at GPT-2 layer 6 while making the IRN sparser (13.9 active features, −81% over standard training; Fig. 5).
- Certified feature retention remains low (2–3 of the top-20), which the authors attribute to the base model's own fragility: it flips 21% of next-token predictions under MINOR paraphrases (Fig. 6).
- Compositional verification of the entire LLM via block-wise containment of propagated sets is proposed as a roadmap (Sec. 6) but is not yet demonstrated.
Strengths
Genuine methodological first with real engineering
Adapts set-based NN verification (CORA, zonotopes) to certify per-layer faithfulness of SAEs/transcoders; the one-step look-ahead collapsing Wdec·diag(m)·Wenc keeps dictionaries up to dIRN=131,072 (Llama Scope) tractable, with Prop. 1 giving O(d²model·dIRN) time and Lean 4-checked soundness (App. B.2).
Exemplary bound-reporting discipline
Every headline quantity is reported as a certified/attack bound pair (δ ≤ δ* ≤ δ̄; J̌ ≤ J* ≤ J̄, App. B.3), explicitly separating what attacks witness (overlap upper bounds) from what verification certifies (lower bounds), including the counterintuitive direction that Tab. 1 numbers are overlap upper bounds.
Breadth beyond the toy setting
Targets a real, under-examined gap
Unusually transparent about negative results
Weaknesses
Certificates are local and the threat model is heuristic
No principled faithfulness threshold
Statistical reporting is descriptive only, with biased target selection
The training fix improves the verifier's bound more than interpretation stability
Unexplained anomalies and single-stack dependence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a real gap: interpretations of LLMs are routinely read off interpretable replacement networks (IRNs) — sparse autoencoders (SAEs) and transcoders — whose faithfulness is evaluated only on clean data. The authors make three claims: (1) IRN feature sets flip under minor semantics-preserving paraphrases across five open-weight families (Sec. 3, Tab. 1); (2) reachability analysis can certify a sound per-layer upper bound δ on the IRN-vs-model faithfulness gap (Sec. 4, Def. 1); (3) verification-aware training (SET, Koller et al. 2025) plus LoRA tightens that certified bound by ≈90% at GPT-2 layer 6 while increasing sparsity (Sec. 5, Fig. 5). The abstract framing — the first formal guarantees for mechanistic interpretability of LLMs — is accurate only under a strict reading: guarantees are per-layer, local, and conditional on a heuristic threat model, and Sec. 6 admits that end-to-end compositional verification is a vision, not a result.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is an IRN and why does its faithfulness matter?
An interpretable replacement network — typically a sparse autoencoder (SAE) or transcoder — stands in for a dense LLM computation and exposes human-readable features; published claims such as a neuron 'firing for the Golden Gate Bridge' are read off these proxies. If the IRN does not faithfully match the model's computation, the resulting interpretation can mislead — for example, a safety auditor reading features to approve or block a request may be reading the wrong signal.
Did the authors actually verify a whole LLM?
No. The results are per-layer certificates: for each transformer block, the difference between the block's computation and its IRN is bounded within a local box of paraphrase-perturbed activations. Propagating one verified input set through the entire model (compositional verification) is presented as a vision and roadmap in Sec. 6, not as an implemented result.
Does verification-aware training make interpretations reliable?
It substantially tightens the certified faithfulness bound (about 90% smaller δ at GPT-2 layer 6) and increases sparsity (13.9 active features vs 74.4 baseline), but the guaranteed number of retained features stays low — 2–3 of the top-20 — largely because the underlying GPT-2 itself changes its next-token prediction for 21% of minor paraphrases. The authors frame this as a step toward verified IRNs, not a fix.
Which models and interpretability artifacts were tested?
GPT-2 (OpenAI ReLU SAE/transcoders), Gemma 2 2B and Gemma 3 1B (Gemma Scope JumpReLU SAEs), Llama 3.2 1B (Llama Scope TopK SAE, dictionary size 131,072), and R1-Distill-Qwen 1.5B (EleutherAI TopK SAE). Attack experiments cover all five; per-layer verification results appear in the appendices, while the main-text verification focuses on GPT-2.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (1, 3–13) are extracted as captions and axis labels only; figure-level values (e.g., per-layer δ curves in Figs. 4–6) could not be checked beyond the in-text claims.
- PDF extraction garbles math notation (subscripts, bound symbols in Alg. 2, Tab. 6/8/9 layout); all numbers cited in this review were cross-checked against their surrounding text, but minor cell misalignment is possible.
- The supplementary material (code, Lean formalizations) is referenced but not part of the extracted text; the reproducibility assessment relies on the paper's stated claims about it.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.