The Endless Exam: Mathematical Constructions from Today's Models toward Superintelligence
cs.AI · math.HO
Quick verdict
Solid open benchmark: 14 construction families, uncapped auto-verified scoring; 0% of tool-free runs beat published frontiers, single-run design weakens stats.
Summary
This paper introduces The Endless Exam, a benchmark of 14 parameterised families of mathematical construction problems (cap sets, spherical codes, Schur partitions, degree-diameter graphs, MOLS, LABS, Heilbronn, Shannon and trifference codes, etc.) instantiated as 69 instances: 30 scored against published frontiers and 39 against construction baselines. Submissions are checked by two independent verifiers and scored by an uncapped relative-quality ratio, so surpassing a reference yields scores above 1. Across eight models in twelve tool-free configurations, overall scores span 7.55 to 91.90, with GPT-6 Astra high producing valid constructions on all 69 instances; no configuration surpasses any published frontier. With code and web access (two-hour, four-thread budget), Astra reaches 143.16 and Luna 119.35, matching 29 and 23 of 30 frontiers and improving 33 and 25 of 39 baselines. Size-quality curves show quality degrading on larger instances. Limitations: one response per instance and configuration, so run-to-run variance is unmeasured; baselines were calibrated after model responses were collected, and for one trifference instance the baseline rests on a construction generated by a model itself.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Uncapped relative quality against references (frontiers or construction baselines) is a sound measure of mathematical construction progress that preserves magnitude of improvements beyond score 1 (Sec. 4.1).
- Compact descriptions plus algebraic certificates allow fully automatic verification of constructions far too large to enumerate, including a Shannon code with over a trillion codewords (Sec. 3.4, B.2).
- Across eight models and twelve tool-free configurations, continuous scores distinguish performance (7.55-91.90 overall) even though no evaluated system surpasses any published frontier on the 30 frontier instances (Table 2, Sec. 5.1).
- Tool access (code + web, 2h) raises GPT-6 Astra to 143.16 and GPT-5.6 Luna to 119.35, matching 29 and 23 of 30 published frontiers without surpassing any, and improving 33 and 25 of 39 construction baselines (Sec. 5.1, H.2).
- Parameterised families and size-quality curves provide a continuing supply of harder instances as models improve, with quality degrading on larger instances (Sec. 5.2, Figures 4, 10-13).
Strengths
Uncapped, verifiable scoring design
Relative quality r(a,p)=a/h(p) (Sec. 4.1) preserves magnitude beyond references; the Schur worked example (Table 4) shows 5,286 vs 5,362 constructions scored 1.049 vs 1.064 against the 5,041 historical reference — a graded signal binary pass/fail benchmarks lose.
Strong verification engineering
Two independent verifier implementations agree on all model constructions and all 69 reference constructions (Sec. 3.4, D.1); algebraic certificates handle a Shannon code with >10^12 codewords (B.2) and rank-based trifference checks with reported timings (Table 14).
Attainability control
Unusual transparency and release depth
Honest headline result
Weaknesses
Single-run evaluation with variance-free inference
Baseline circularity and post-hoc calibration
Protocol heterogeneity across models
Tool-track gains conflate tools with a different budget
Self-serving comparison table and aspirational framing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: The Endless Exam (arXiv 2609.24555)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Endless Exam benchmark?
It is a benchmark of 69 mathematical construction problems drawn from 14 parameterised families (cap sets, spherical codes, Schur partitions, degree-diameter graphs, codes, and others). AI models must produce valid constructions, which are checked automatically by two independent verifiers and scored by an uncapped ratio against a published-best reference or an author-established baseline; scores above 1 mean the reference was beaten.
Did any AI model beat the best known mathematical constructions on the Endless Exam?
No. In tool-free mode, none of the twelve evaluated configurations surpassed any of the 30 published-frontier references (0% breakthrough rate). With code execution and web access, GPT-6 Astra matched 29 of 30 published frontiers without exceeding any, though it did improve on 33 of 39 self-established construction baselines.
How are huge constructions with trillions of elements verified?
Instead of listing every element, models can submit compact descriptions such as products of small verified codes or generator matrices. The verifier checks the small ingredients and algebraic conditions, then computes the exact size — for example, a Shannon code with over a trillion (10^12) codewords was certified from finite factors, and a 10x64 generator matrix certifies 3^10 trifference codewords.
Which models were evaluated and what were the scores?
Eight models in twelve tool-free configurations: overall scores ranged from 7.55 (Qwen3.5-4B) to 91.90 (GPT-6 Astra at high effort, 100% valid responses). With code and web access, GPT-6 Astra reached 143.16 and GPT-5.6 Luna 119.35 on the same 69 instances.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are extracted as unlabeled text fragments, so visual results (Figures 2-5, 7-13) were assessed only via their captions and tables.
- Appendix H's Table 25 is truncated at row 51 (the trifference n=64 row) in the extraction; remaining rows inferred consistent but not directly read.
- Scores depend on proprietary API models; the benchmark assets are reproducible but the exact evaluations are not re-runnable by third parties.
- The text ends mid-table after page 51; any content on later pages was not reviewed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.