Stable Answers, Unfinished Reasoning: Why Self-Consensus Is Not a Safe Early-Exit Signal
Yunxiang Mo, Donghao Zhao, Hejia Geng
cs.CL
Quick verdict
Preregistered sweep: no safe+saving consensus rule (0/3,520) while DEER clears all gates; rigorous negative result, scoped to competition math only.
Summary
The paper asks whether probe-based self-consensus—repeatedly querying a single partial reasoning trajectory for its current answer and stopping once probes agree—is a safe early-exit signal. On frozen trajectories from two distilled models (DeepSeek-R1-Distill-Qwen-7B, Qwen3-8B) across three competition-math benchmarks (MATH500, AMC23, AIME24), yielding 18 model×benchmark×seed environments, the authors preregister a sweep of 3,520 windowed-consensus rules with three fixed acceptance gates (accuracy drop ≤1.0/2.0/3.5 pp paired with token saving ≥10/20/30%). No rule passes any gate; 478 rules pass in-sample on train but none is selectable from dev, while the frontier reproduces on the held-out test split (r=0.98) and on two unseen models (r=0.97 and 0.94). A boundary-confidence control, DEER, swept through the identical pipeline, clears all three gates (−0.33 pp at 28.2% saving). Mechanistic measurements attribute the failure to a consensus–termination gap: two probe wordings disagree 54% of the time early in trajectories versus 16% near the end; 56.7% of 134 hand-labelled wrong stops commit unconverged placeholder answers; at a rule saving 32%, 10.6% of stops commit answers the trajectory later abandons (155 pre-empted corrections versus 16 banked correct answers), and wider windows floor near 7% non-terminal stops only as saving falls to 8%. Limitations include restriction to competition math, two model families, very small AMC23/AIME24 dev cells, and savings figures that presuppose KV-cache reuse because probe prefill is uncharged.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Repeated probe agreement on a single trajectory establishes only that the current answer persists under a fixed probing procedure, not that reasoning has terminated — the consensus–termination gap (§4).
- No rule in a preregistered family of 3,520 windowed-consensus rules is simultaneously safe and token-saving on the dev split (0/3,520 pass any gate), with the frontier reproducing on the test split (r=0.98) and two unseen models (r=0.97/0.94).
- Early consensus answers are largely probe-elicited placeholders: two probe wordings disagree 54% of the time in the first tenth of trajectories, and 56.7% of 134 hand-labelled wrong stops commit answers the model had not converged on.
- Widening the agreement window trades saving for safety: non-terminal stops only floor near 7% while net saving falls to 8% (§4.4, Table 7), and a stop at a 32%-saving rule commits a non-terminal answer once in nine, pre-empting a correction in most such cases.
- A boundary-confidence control (DEER trial-answer-submit) swept through the identical pipeline clears all three preregistered gates (−0.33 pp at 28.2% saving on dev), so early exit remains viable and the consensus statistic, not early exit, is what fails (§5.3, Table 3).
Strengths
Preregistered, commitment-based evaluation
Gates, splits and the hashed rule set were fixed before any held-out read, with three explicit commitments C1–C3 (App. B); an empty gate is reported as the finding rather than tuned away — rare discipline, especially for a negative result.
Frozen-trajectory offline probing isolates the signal
§3.1 freezes one trajectory per problem and performs all probes offline, so no rule can change the reasoning it observes and all 3,520 rules are scored on identical substrates; §5.6/Table 3 then separates the statistic from the read positions, showing only the statistic moves the accuracy axis.
Mechanistic triangulation, not just a benchmark verdict
Non-consensus control under identical protocol
Exceptional accounting and reproducibility hygiene
Weaknesses
Narrow domain and model diversity
Headline margin is sensitive to the aggregation/floor choice
Very small evaluation cells drive the binding constraints
Mechanism evidence is observational and regime-specific
Savings presuppose infrastructure not always available
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a negative-result empirical study of early-exit signals for reasoning LLMs. The authors claim that self-consensus — repeatedly appending an answer probe to one partial chain of thought and stopping when recent probes agree — cannot be simultaneously safe and token-saving, and that the failure is mechanistic rather than a tuning artifact: agreement certifies that the current answer *persists under a fixed probe*, not that reasoning has *terminated* (the consensus–termination gap, §4). Three claims carry the paper. First, a preregistered sweep of 3,520 windowed-consensus rules over 18 frozen environments (two distilled models × MATH500/AMC23/AIME24 × three seeds) clears none of three fixed gates (drop ≤1.0/2.0/3.5 pp paired with saving ≥10/20/30%; §5.2). Second, three direct measurements show early agreement is a probe-elicited placeholder (§4.2–4.4). Third, a non-consensus control, DEER, swept through the identical pipeline clears all gates (§5.3), isolating the signal — not early exit itself — as the culprit. Scope is explicitly competition mathematics with checkable answers, one probe suffix and one rule family, as the Limitations section states plainly.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the consensus–termination gap?
It is this paper's term for the mismatch between what probe agreement actually shows and what early exit needs. Repeatedly probing a partial reasoning trajectory with the same question shows only that the currently elicited answer persists under that fixed probe; it does not show the reasoning has finished. Early probes often force out a placeholder answer that repeats across many probes while the trajectory would later correct itself — so stopping on agreement locks in unfinished answers.
Does this mean early exit from reasoning models is impossible?
No. The same pipeline, gates and token accounting show that DEER — which reads the model's confidence at reasoning boundaries rather than answer agreement — clears all three preregistered gates, losing as little as 0.33 pp accuracy while saving about 28% of tokens. The paper's negative result is specific to answer-agreement (self-consensus) signals used alone on a single trajectory.
Does this refute Dynasor / CertaIndex (Fu et al., 2024)?
Not the multi-path version. The paper reproduces only CertaIndex's single-chain-of-thought stopper at its shipped defaults, which stops on 98.7–99.8% of trajectories and loses 56–70 pp on these competition-math benchmarks. The authors explicitly note that CertaIndex's multi-path setting resembles self-consistency over independently sampled trajectories and is untouched by their results. They also argue the original CoT results were reported on an easier benchmark where answers settle sooner.
Which models and benchmarks were used, and do the findings generalize?
Development used DeepSeek-R1-Distill-Qwen-7B and Qwen3-8B on MATH500, AMC23 and AIME24 (18 model×benchmark×seed environments), with confirmation on a held-out test split and two unseen models (DeepSeek-R1-Distill-Llama-8B and Distill-Qwen-32B). The trade-off frontier reproduces across splits and models, but everything is competition mathematics with checkable answers — the paper does not claim the gap transfers to open-ended reasoning, code or agentic tasks.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with [...sampling gap...] regions (parts of §2 related work, the middle of §5.1 including Table 1's body, parts of Appendix D and the Case-2 study in Appendix F, and chunks of the reference list); all numbers cited in this review come from the provided portions, and gate values are taken from the Figure 1 caption and §5.2/App. B text rather than the (unseen) Table 1 body.
- Figures 1–9 appear in the extraction only as flattened caption and axis text; quantitative statements rely on in-text numbers and table captions rather than rendered plots.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.