Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction
cs.CL · cs.LG
Quick verdict
Preregistered reproduction confirms the CoT entropy-shape signal on 4 models but finds the magnitude null setting-dependent; rigorously, candidly reported.
Summary
This preregistered study (OSF 8w2q3) independently reproduces Zhao (2026)'s dissociation: chain-of-thought entropy trajectory shape (epsilon-monotone decline) predicts answer correctness, while magnitude (total entropy drop) supposedly does not. Four open-weight 7-8B models (Qwen2.5-7B-Instruct as anchor, Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct, DeepSeek-R1-Distill-Qwen-7B) were run over the complete GSM8K (n=1319) and MATH-500 test sets, yielding 7,013 usable records across eight confirmatory cells. The shape signal replicates: anchor gaps of +9.62 (GSM8K) and +27.49 (MATH-500) percentage points, with all four secondary standard-model cells positive; the graded violation count holds in all eight cells. The magnitude null fails broadly: it meets criterion in only 2 of 8 cells, with rho=+0.414 on anchor MATH-500, contradicting the original. On the reasoning-distilled model the binary flag fires on ~1% of chains, making the registered contrast unestimable. Exploratory analyses show final-step entropy alone beats the binary flag by ROC area in all eight cells, suggesting the informative quantity is where trajectories end, not monotone descent. Limitations: reference-chain temperature 0.7 versus the original's 0.1 with the convergence route untested, a self-described segmentation-rule defect causing up to 14% exclusions, single-seed operation with the pre-specified three-seed check unrun, and part of the analysis code still in a private repository.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 0 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The binary entropy-shape (monotonicity) signal replicates at full test-set scale on three standard instruct models across GSM8K and MATH-500, under seven documented protocol differences from the original.
- The scalar-coherence (magnitude) null reproduces only in the single cell matching the original's measurement conditions and fails the registered criterion in most other cells, e.g., rho=+0.414 on anchor MATH-500.
- The graded violation count predicts correctness in all eight cells, including on the reasoning-distilled model where the binary flag collapses.
- On DeepSeek-R1-Distill-Qwen-7B the binary monotonicity flag fires on about 1% of chains, making the registered H4 contrast unestimable and showing the method's applicability is bounded by chain properties.
- Exploratorily, final-step entropy alone beats the binary shape flag by ROC area in all eight cells, suggesting the informative quantity is where the trajectory ends rather than monotone descent.
Strengths
Genuine preregistration with honestly scoped hypotheses
OSF registration 8w2q3 (2026-08-29) preceded every confirmatory run; Section 3.4 and Appendix C state the primary/secondary scope (H1/H4 primary) rather than inflating post hoc, and the unregistered 16-test family of an earlier revision is disclosed
Census-scale panel with exact comparison axis
Complete GSM8K (1319) and MATH-500 across four models gives 7,013 usable records; Section 2 correctly benchmarks against Zhao's full-scale numbers (+11.5/+33.3 pp) rather than his 300-problem pilot (+21.9 pp)
Exceptional transparency about deviations and failures
Diagnostics that kill candidate artifacts
Exploratory/confirmatory boundary maintained
Weaknesses
Most consequential parameter deviates by accident and the key route is untested
Registered segmentation rule is self-described as defective
H5 confirmatory verdict is a post-hoc recovery
Single-seed results with material run-to-run instability
Secondary analysis code still private
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a preregistered, independent reproduction of the dissociation reported in Zhao (2026): that the *shape* of a chain-of-thought answer-entropy trajectory (epsilon-monotone decline, epsilon=0.01) predicts final-answer correctness, while its *magnitude* (scalar coherence C = H0 - HN) does not. Five hypotheses were registered at OSF (8w2q3, 2026-08-29) before any confirmatory run. Four open-weight 7-8B models (Qwen2.5-7B-Instruct as anchor, Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct, DeepSeek-R1-Distill-Qwen-7B) crossed the complete GSM8K (n=1319) and MATH-500 test sets, yielding 7,013 usable records over eight confirmatory cells. The claims are deliberately scoped: a full-test-set confirmation of the shape signal, a division of the magnitude claim into a violation half and a scalar-coherence half, and measurements of four protocol dependencies the original never reports.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 0 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What did this reproduction find about Zhao's entropy-trajectory claim?
The paper preregistered (OSF 8w2q3) and ran four open-weight 7-8B models over the complete GSM8K and MATH-500 test sets. The shape signal - chains whose answer-entropy falls monotonically are more often correct - replicated on all three standard instruct models. The magnitude signal - total entropy drop being uninformative - replicated only in the one condition matching the original's setup and failed the registered criterion in six other cells, most sharply with rho=+0.414 on Qwen2.5-7B MATH-500.
Which models and datasets were used in the study?
Four open-weight models at pinned Hugging Face revisions: Qwen2.5-7B-Instruct (anchor, matching the original), Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct, and DeepSeek-R1-Distill-Qwen-7B (a reasoning-distilled model the original did not test), each run over full GSM8K (1,319 problems) and MATH-500 (500 problems).
Is the study preregistered?
Yes - OSF registration 8w2q3, dated 2026-08-29, before any confirmatory run, with the analysis plan frozen in a public repository as tag prereg-v1.0 and a SHA-256 recorded in the paper. Five hypotheses were registered, with H1 and H4 designated primary.
What are the main caveats about this reproduction?
The reproduction differs from the original in seven documented ways, most notably reference-chain temperature 0.7 versus the original's 0.1 and a blank-line segmentation rule the author himself calls defective (it conflicts with the study's own prompt and excluded up to 14% of Mistral chains). All results are single-seed. The author frames the confirmation as a statement about robustness rather than pipeline identity.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (coverage curves) is assessable only via its caption and surrounding text, not visually
- The abstract in the prompt is truncated mid-sentence, but the full text through Appendix K and references was provided
- Appendix tables were extracted as plain text; column alignment was inferred and could hide minor extraction errors
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.