Euston: Training Away Mathematical Sycophancy Without Losing the Mathematics
Zehua Cheng, Wei Dai, Jiahao Sun
cs.CL
Quick verdict
Trains 8B model to reject false math (+27.5pp gap) claiming AIME retained; notably candid, but attribution rests on recalled ablation numbers.
Summary
The paper introduces Euston, an 8B model (fine-tuned from DeepSeek-R1-0528-Qwen3-8B with GRPO, 189 steps, four H100s, ~13.8h) trained to reject mathematically false statements instead of confidently proving them. Training data come from GraphSynth, the authors' factor-graph generator, producing 3,026 matched true/corrupted statement pairs (6,052 statements) from arXiv 2010–2025, with a rule-based zero-API reward on the final boxed verdict. On a balanced 200-true/200-false held-out split, balanced accuracy rises from 29.50% to 63.75%, and the discrimination gap from −0.5pp (z=−0.1) to +27.5pp (z=+6.0); roughly 18pp of the gain reflects emitting a parseable verdict at all (truncation fell from 38% to 0.25%) and ~16.3pp improved judgement. AIME 2026 under official semantics drops non-significantly (69.17%→65.00%), though landed-runs-only scoring shows a significant loss (−12.96pp); an earlier smaller-corpus run collapsed to 40.00%. Limitations the authors flag: official BrokenArXiv sets are all-false (inflating disprove gains), false-rejection of true statements rose to 51.5%, precision at realistic ~1% error prevalence is ~1.5%, the pilot contrast is not a controlled ablation, Section 5.6 baseline values are approximate recalled figures, and Euston was selected after seeing evaluations.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GRPO fine-tuning on 3,026 GraphSynth true/corrupted pairs raises the discrimination gap on a balanced 400-item split from −0.5pp (z=−0.1) to +27.5pp (z=+6.0).
- The discrimination gain does not significantly degrade general math ability under official AIME 2026 semantics (65.00% vs 69.17%, n.s.), whereas an earlier run of the same recipe on a smaller corpus collapsed to 40.00%.
- Roughly half of the 34.25pp balanced-accuracy gain (~18pp) reflects emitting a parseable verdict at all rather than improved judgement (~16.3pp).
- Whether discrimination is bought at the cost of capability is determined by the quality and volume of generated negatives rather than the optimizer — an attribution currently supported by approximate recalled baselines and a non-controlled pilot comparison.
Strengths
Balanced, shortcut-resistant evaluation design
The 200-true/200-false held-out split (Sec 4) makes the constant-'false' strategy score exactly chance (50%), and the discrimination gap P(false|false)−P(false|true) is immune to any constant shift in skepticism — the right headline metric, reported with z-scores (−0.1 → +6.0).
Honest decomposition of the headline gain
Sec 5.1 splits the +34.25pp balanced-accuracy gain into ~18pp from emitting a parseable verdict (truncation 38.0%→0.25%) and ~16.3pp from judgement, and prominently reports the adverse false-rejection rise on true statements (38.5%→51.5%) rather than burying it.
Transparent treatment of confounds and scoring bounds
Strong reproducibility specifics
Temporally and textually disjoint evaluation
Weaknesses
Central attribution rests on unmeasured baselines
Pilot contrast is not a controlled ablation
Retention claim is bound-dependent on a tiny sample
Official-protocol evidence is thin and partially non-significant
Practical utility is bounded by false rejection
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Euston (arXiv 2609.23205, cs.CL) is a technical report describing an 8B claim-verification model: DeepSeek-R1-0528-Qwen3-8B fine-tuned with GRPO (verl) for 189 steps on 4×H100 (~13.8h) against a rule-based, zero-API reward (1.0 iff the final \boxed{} verdict casefold-matches the label, Table 1). Training data are 3,026 matched true/corrupted statement pairs (6,052 statements) from arXiv 2010–2025, produced by the authors' GraphSynth factor-graph generator (Cheng et al., ACL 2026). Four claims are made: (1) discrimination on a balanced 400-item held-out split improves sharply (gap −0.5pp → +27.5pp, z=+6.0); (2) general capability is retained under official AIME 2026 semantics (65.00% vs 69.17%, n.s.) where an earlier smaller-corpus run of the same recipe collapsed to 40.00%; (3) roughly half the balanced-accuracy gain is parseability, not judgement; (4) generated-data quality, not the optimizer, decides whether discrimination costs capability.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is mathematical sycophancy in language models?
It is the tendency of reasoning models to comply with a false premise: asked to prove a corrupted theorem, the model confidently derives something untrue instead of flagging that the statement is false. The Euston paper trains an 8B model to resist this by emitting a True/False verdict on mathematical claims taken from arXiv papers.
Does Euston lose general math ability compared to the base model?
Under official AIME 2026 scoring (dropped runs score zero), the drop from 69.17% to 65.00% across 30 problems is not statistically significant. However, counting only runs that produced an answer, Euston is significantly worse (71.26% vs 89.51%), partly because the base model truncates far more often (25.8% vs 8.3%), so the verdict depends on which scoring bound you accept.
Why is the 63.75% balanced-accuracy number misleading?
Because roughly 18 of the 34.25-point improvement comes from the model terminating with a parseable True/False verdict at all — the base model failed to answer 38% of items within budget — rather than from better judgement, which accounts for about 16.3 points. Only the second component is what the benchmark intends to measure.
Can Euston be used to screen real papers for mathematical errors?
Not effectively yet. It rejects 51.5% of true statements, and at a realistic 1% error prevalence only about 1.5% of its 'this is false' flags would be correct, because a +27.5pp discrimination gap cannot overcome a 99:1 prior favoring correct statements.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as text fragments in the extraction; visual details of Figures 1–4 (e.g., the capability–discrimination plane, hatched approximate bars) could not be independently analyzed.
- Table 1 and several tables were extracted from inline text with minor formatting artifacts; all quoted numbers were cross-checked against the prose.
- References to GraphSynth (ACL 2026) and the BrokenArXiv benchmark could not be verified from this text alone; the paper is a non-peer-reviewed technical report.
- Section 5.6 reports hatched/approximate values reconstructed from logs — the review reflects the authors' own flagging of these as direction-only evidence.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.