Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning
Kevin Du, Alexander Hoyle, Laura Ruis, Acyr Locatelli
cs.CL · cs.LG
Quick verdict
CoT step importance as MC-estimated advantage: judges and critics decode it far below noise ceiling—a rigorous negative result, limited to math and Qwen models.
Summary
The paper asks whether the text of a chain-of-thought (CoT) step encodes how much that step actually mattered to the model's answer. It operationalizes step importance as the reinforcement-learning advantage: the change in expected reward — matching the trace's own final answer (self-advantage, the default) or correctness — caused by appending the step, estimated with 50 Monte Carlo rollouts per reasoning step of Qwen3-1.7B/4B/8B (thinking off, plus a 1.7B thinking arm) on 180 problems from six math benchmarks (AIME 24/25/26, AMC 23, MATH500, GSM8K). A calibrated changepoint pipeline (PELT plus Beta-posterior test at |Δp|>0.1 with ≥0.95 probability) labels steps consequential. Out-of-the-box judges (Qwen3-1.7B up to Qwen3.6-27B) beat the prevalence baseline with scale but sit 6–9× below split-half noise ceilings in PR-AUC. LoRA fine-tuned critics reach roughly half the ceiling on incorrect responses (0.28–0.30 ID) but only 10–20% of it on correct responses (0.065–0.10), an asymmetry the authors attribute to limited decodability of importance in correct solutions, possibly inflated on the incorrect side by easy final-answer-emission proxies. Descriptive analyses find uncertainty-management steps most often consequential, and cue prompting collapses consequential responses from 58% to 15%. Limitations: math-only tasks, a single (Qwen) model family for generators and judges, and gold labels that are themselves noisy Monte Carlo estimates.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Step importance in a chain-of-thought can be operationalized as the RL advantage of the step, measurable via Monte Carlo rollouts with changepoint-based labeling at controlled false-discovery rates.
- Out-of-the-box LLM judges beat a prevalence baseline and improve with scale, but the best judge (Qwen3.6-27B) remains 6–9× below the split-half noise ceiling at identifying consequential steps.
- Fine-tuned critics decode step importance to roughly half the ceiling on incorrect responses (PR-AUC 0.28–0.30 ID) but recover only 10–20% of the ceiling on correct responses (0.065–0.10).
- Step importance is therefore only partially recoverable from CoT text, cautioning against practices — LLM-judge evaluation and process reward modeling — that read trace text as interpretable.
- Descriptively, uncertainty-management steps are most often consequential for correct answers, and adding a cue collapses consequential responses from 58% to 15%, suggesting cues determinize reasoning.
Strengths
Principled formalization of step importance
§3 defines value, Q-value, and advantage over string states, explicitly contrasting self-advantage (matching the trace's own answer) with correctness-based reward; Appendix K proves A_LOO = A/(1−πϕ(a|s)), connecting the measure to counterfactual-importance alternatives like Bogdan et al. (2025).
Statistical pipeline with calibrated error control
Appendix A.1 calibrates the PELT penalty per (model, mode, dataset) cell on 1,000 simulated flat null series to a 2% per-series FPR, gates boundaries with a Beta-posterior effect-size test (P(|Δp|>0.1)≥0.95, δ=0.1), and reports a posterior expected FDR ≈0.2% over the 7,563 consequential steps; Appendix A.3 compares against Benjamini–Hochberg per-step testing.
Noise-ceiling discipline in evaluation
Honest, well-ablated negative result
The measure doubles as an analysis tool
Weaknesses
Narrow external validity
Unexplained flat critic scaling
Arbitrary step segmentation
Headline reward is the more decodable one
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This COLM 2026 paper (arXiv 2609.04194) asks whether chain-of-thought (CoT) text encodes which steps matter. It operationalizes step importance as the RL advantage A(s,a)=Q(s,a)−V(s) (§3): the change in expected binary reward — matching the trace's own final answer ('self-advantage', the default) or correctness — caused by committing to step a from prefix s, estimated with N=50 Monte Carlo continuations per prefix. Steps are labeled consequential when a changepoint pipeline (PELT; §4, App. A) certifies |Δp|>0.1 with ≥0.95 posterior probability. Three claims are tested: (1) advantage is a useful analytical lens (§5); (2) out-of-the-box judges only weakly identify consequential steps (§6, Fig. 6); (3) even fine-tuned critics recover far more signal on incorrect than correct responses, so importance is only partially decodable from text (§6–7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'legibility is not interpretability' mean in this paper?
The paper argues that a chain-of-thought trace can read clearly and logically (legible) without its text actually encoding which steps functionally mattered to the answer (interpretability). The authors measure each step's true causal impact via Monte Carlo resampling and show that LLM judges, and even fine-tuned critics, recover only a small fraction of that signal from the text alone — especially in responses that end in a correct answer.
How did the authors measure the true importance of a reasoning step?
They define importance as the reinforcement-learning advantage of a step: the change in the probability that the model reaches the trace's final (or the correct) answer when that step is included, estimated by sampling 50 continuations from the prefix before and after the step. A changepoint analysis with a calibrated statistical test marks steps as 'consequential' when the probability jump exceeds 0.1 with at least 95% posterior probability.
Can LLM judges identify which reasoning steps are important?
Only partially. Out-of-the-box judges improve with model scale and beat a prevalence baseline, but the best judge tested (Qwen3.6-27B) still scores 6–9 times below a noise ceiling that accounts for label noise. Fine-tuned critics do much better on responses ending in wrong answers (about half the ceiling) but recover only 10–20% of the ceiling on correct responses.
What are the implications for process reward models (PRMs)?
PRMs and step-level critics assume step text carries signal about a step's contribution. This paper suggests that assumption is only weakly true: text-based predictors approach the noise ceiling on incorrect responses but stay far from it on correct ones. Step-level supervision learned purely from trace text may therefore miss much of the true step-level signal, particularly for successful reasoning — and the paper's correctness-based reward results (Appendix J) are even weaker.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Provided text is a sampled excerpt: several [...sampling gap...] ranges skip content (e.g., parts of §5's value-trajectory analysis and Appendix A.1's localization detail, plus portions of appendix sections D.2/F.1/H.2); judgments rest on the visible core, and the tail is complete, ending at §M (LLM Usage Statement) with no end truncation.
- Figures were extracted only as partial axis/legend text; quoted curve values come from the running text (e.g., §6, B.2, B.3) rather than from reading the plots.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.