Magenta: Closing the Loop Between Mathematical Reasoning and Lean Verification
Joshua Ong Jun Leang, Haonan Li, Zheng Zhao, Xinyi Shang, Wenda Li, Zhengzhong Liu, Eric Xing, Shay Cohen, et al.
cs.AI
Quick verdict
Magenta: Lean-loop pipeline claims 100% on AIME/HMMT and 6/6 IMO 2026; impressive, but tiny saturated benchmarks, LLM judges and thin IMO scoring temper it.
Summary
Magenta is a training-free agentic pipeline coupling informal LLM mathematical reasoning with Lean 4 verification. A reasoner produces a solution and candidate answer; a formaliser converts problem plus answer into a Lean 4 statement; a statement judge (an LLM) checks the formalisation preserves the original problem; a prover builds a machine-checked proof; and an error judge labels failures as SYNTAX (local proof repair) or MATH (re-derivation with feedback), while SafeVerify blocks sorry, admit, new axioms and native decide. On AIME 2025, AIME 2026 and HMMT February 2026, the pipeline verifies 100% of problems with four reasoners; paired with open-weight K2-Horizon-7B it solves all six IMO 2026 problems, scored by a CODEX judge against public reference solutions plus one former IMO medallist. Ablations show removing the statement judge drops verified-correct rate from 63.3% to 20.0% with a 45.5% false-certification rate (Goedel formaliser, AIME 2026), and diagnostic-conditioned correction reaches 6/6 on IMO where resampling gets 1/6. Paraphrased AIME 2026 costs standalone reasoners 3.3-10 points while Magenta stays at 100%. Limitations: certificates are only 'soft' - sound relative to an LLM-adjudicated statement; benchmarks are small and saturated; IMO scoring lacks inter-rater statistics; and concrete budgets plus any code-availability statement are not visible in the provided text.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A training-free agentic pipeline can turn a natural-language problem into an answer plus a machine-checked Lean 4 proof, with an LLM statement judge guarding formalisation faithfulness and an error judge routing failures to proof repair or re-derivation.
- Magenta verifies 100% of AIME 2025, AIME 2026 and HMMT February 2026 with four reasoners, and with open-weight K2-Horizon-7B solves all six IMO 2026 problems — claimed the smallest reported reasoner to do so.
- Statement adjudication is essential: disabling it on AIME 2026 (Goedel formaliser) drops VerCor from 63.3% to 20.0% and yields a 45.5% false-certification rate.
- Diagnostic-conditioned self-correction outperforms independent resampling on hard problems: 6/6 vs 1/6 on IMO 2026 under the paper's budget, and 30/30 vs 29/30 on AIME 2026.
- Magenta is robust to answer-preserving paraphrases of AIME 2026 (Δ=0) where standalone reasoners lose 3.3–10.0 points.
Strengths
Targets the actual reliability bottleneck
§2.3 and Table 3 quantify the autoformalisation gap: without the statement judge, VerCor falls 63.3%→20.0% and FCR reaches 45.5% on AIME 2026 (Goedel formaliser); Table 6 shows the judge rejects 67.08% of Lean-passing statements, i.e. compilation is a weak faithfulness proxy.
Error attribution is a real mechanism, not a retry loop
§4.3 routes SYNTAX failures to local prover repair and MATH failures to re-derivation; on IMO 2026 diagnostic-conditioned correction verifies 6/6 vs 1/6 for 8–16 independent resamples, and the single AIME 2026 resampling failure (unprovable statement from a wrong answer) is explained.
Certification hygiene
Honest scope and contamination controls
Budget-aware, non-cherry-picked efficiency analysis
Weaknesses
Headline rests on tiny saturated benchmarks
LLM judges sit inside the trust chain and are never calibrated
Thin IMO evaluation protocol
No statistical treatment
Weak reproducibility signal in the provided text
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Magenta (arXiv 2609.11319v2) is a training-free agentic pipeline mapping a natural-language competition problem to (i) an informal reasoning chain and answer, (ii) a Lean 4 statement of problem plus answer, and (iii) a machine-checked proof. Five components: reasoner R, formaliser F, Lean verifiers Vs/Vp augmented with SafeVerify, statement judge Js, and error judge Je (SYNTAX → local prover repair; MATH → re-derivation with feedback). Central claims: 100% verified accuracy on AIME 2025, AIME 2026 and HMMT February 2026 with four reasoners; all six IMO 2026 problems solved with open-weight K2-Horizon-7B (claimed smallest reasoner to do so); statement adjudication prevents false certificates; diagnostic-conditioned correction beats resampling.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Magenta (the AI system for math)?
Magenta is a training-free agentic pipeline that connects LLM mathematical reasoning with Lean 4 formal verification: an LLM reasons about a natural-language problem, another LLM converts the problem and proposed answer into a Lean 4 statement, a judge checks the formalisation is faithful to the original problem, a prover writes a machine-checked proof, and an error judge routes verification failures to either local proof repair or mathematical re-derivation.
Did Magenta solve IMO 2026?
The paper reports that Magenta paired with the open-weight K2-Horizon-7B reasoner produced verified solutions to all six IMO 2026 problems, scored by a CODEX judge against public reference solutions and independently marked by a former IMO medallist. These are paper claims on six problems with their own scoring protocol, not an official IMO result.
Why isn't a Lean proof check a full guarantee that the answer is correct?
Lean only certifies that the proof establishes the generated formal statement. If that statement mis-translates the original natural-language problem (the 'autoformalisation gap'), the certificate proves something other than what was asked. Magenta's statement judge is itself an LLM, so the paper describes its output as a 'soft certificate'.
What did the statement-judge ablation show?
On AIME 2026 with the Goedel formaliser, disabling the statement judge dropped the verified-and-correct rate from 63.3% to 20.0%, and 45.5% of returned certificates proved something other than the problem asked — mostly weakened or answer-baked surrogate goals (Appendix E.1).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with several [...sampling gap...] regions; §3.1 (main experimental setup: concrete budgets T/M/K and the four reasoner identities) and parts of Appendices C–D were not visible, so budget- and setup-dependent judgments are provisional.
- No code or data availability statement was visible in the provided excerpt; whether one exists elsewhere in the paper could not be verified.
- HMMT February 2026 problem counts and the identities of all four reasoners were not visible in the excerpt.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.