Legible Failures: Detecting and Repairing In-Context Binding Errors
Manas Venkata Sai Ravulapalli, Samrath Singh Chadha, Abhinav M. Hari
cs.LG
Quick verdict
On a synthetic binding task, wrong LLM answers often keep a linearly readable correct binding; solid evidence for detection and repair, scope is synthetic.
Summary
The paper asks whether language models that answer incorrectly still retain the correct answer linearly in their hidden states — “legible failures” — on a synthetic entity–obligation binding task (eight declarations, a distractor block, a single-token query). Across 16 public checkpoints (410M to 14B parameters), each evaluated with three seeds, the authors fit multinomial logistic probes on a training fold, select the read layer on a validation fold, and report on a disjoint test fold. On the trials each model gets wrong, probe accuracy exceeds the 1/K = 0.125 present-set baseline by +0.196 (95% CI [+0.101, +0.296], model-clustered bootstrap; 14 of 16 models pass a pre-set 30-wrong-trial floor). A query-entity counterfactual (+0.371) rules out token presence and recency. A signed probe–output disagreement score improves error detection over model confidence by +0.079 AUROC (CI [+0.036, +0.126]); raw probe confidence adds nothing, and the advantage over predictive entropy is not significant. Self-gated steering toward the probe's decode raises accuracy on all eight intervention models by +0.168 (CI [+0.066, +0.280]) against matched-norm random and gold-oracle controls. Main limitations: the task is synthetic with single-token answers; checkpoint revisions were not pinned; the repair experiments use a different configuration (K=6, D=256) than the legibility sweep; and the prompt-repair battery lacks per-trial records.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On the test trials a model answers wrongly, the correct prompt-supplied binding remains linearly decodable from the frozen hidden state, exceeding the 1/K = 0.125 present-set baseline by +0.196 on average across 14 models (§3, Table 3).
- A query-entity counterfactual shows the probe tracks the query-conditional binding rather than token presence or recency (+0.371 margin; §3.2).
- A signed probe–output disagreement score detects failures better than model confidence (+0.079 AUROC) and sampled self-consistency (+0.065), while raw probe confidence adds nothing over confidence (§4).
- Gold-label-free steering toward the probe decode raises accuracy on all eight intervention models (+0.168 mean), making in-context binding a setting where probe-detected errors are actionable, unlike prior reports (§5).
Strengths
Leak-free probe protocol with verifiable selection behavior
§2.2: probe fitted on a 200-trial train fold, layer chosen on validation, reported on a disjoint test fold with a refit on train+validation; the validation-chosen layer (0.49–0.92 of depth) moves across seeds on 10/16 models, exactly as an honest selection step should
Confound-targeted counterfactual control
§3.2 holds declarations bit-identical and varies only the queried entity, so presence/recency heuristics score 1/K by construction; margin +0.371 [+0.243, +0.503], and the re-implementation (§3.3) shows the decode tracks the new entity (0.733–0.866) while the original obligation falls to 0.095–0.132
Negative and boundary results reported prominently
Statistics match the generalization claim
Independent shared-nothing replication plus intervention controls
Weaknesses
External validity bounded to a synthetic single-token task
Detector's practical advantage over free baselines is unproven
Configuration mismatch between detection and repair claims
Reproducibility friction
Wide heterogeneity behind the average
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether, when a language model answers a controlled entity–obligation binding task incorrectly, the correct prompt-supplied binding remains linearly recoverable from its frozen hidden state — a “legible failure” (§1). Three claims are tested: (1) such failures are common enough to measure (§3), (2) probe–output disagreement detects them better than the model's own confidence (§4), and (3) a gold-label-free steering intervention repairs them (§5). The stated contrast is with recent reports that probe-detected errors resist intervention (refs [13, 2]). Scope is explicitly bounded: synthetic task, single-token answers, white-box access, frozen checkpoints (§7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a “legible failure” in LLMs?
A trial where the model emits a wrong answer, but a simple linear probe reading its internal hidden state can still recover the correct answer that was supplied in the prompt. On a synthetic entity–obligation binding task, the paper finds this happens often enough that probe accuracy on wrong trials beats a 1/K baseline by +0.196 on average across 14 of 16 tested models.
Can activation steering fix LLM mistakes without knowing the right answer?
In this paper's setting, yes with caveats: steering the residual stream toward the model's own probe-decoded binding (no gold label) raised accuracy on all eight tested models by +0.168 on average. This holds only for the synthetic single-token binding task with white-box access; the authors do not claim it transfers to naturalistic tasks, and in their own re-implementation the intervention failed at a fixed site on four small models.
How much better is the probe-based error detector than just using the model's confidence?
The signed disagreement score improves AUROC by +0.079 over model confidence (95% CI [+0.036, +0.126]). But it adds only +0.025 over predictive entropy with a confidence interval crossing zero, and raw probe confidence adds nothing (+0.003) — so the practical gain over simple probe-free baselines is not established.
Is the code or data for this paper available?
The experiments use public Hugging Face checkpoints and a synthetic task, but the paper text gives no repository URL; it references a release pipeline and committed run summaries, and notes that checkpoint revisions were not pinned. The prompt-repair battery is reproducible only at run granularity because per-trial records were not retained.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 appear in the extraction only as embedded text fragments (axis labels, bar annotations); every figure-derived number cited in this review was cross-checked against the body text and Tables 2, 3, 5 and 6, and no numeric content was found missing. The extracted text runs complete through Appendix F and Table 7.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.