RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?
Mithil Salunkhe, Haochen Ding, Samridhi Verma, Volodymyr Kindratenko
cs.AI · cs.LG · cs.SE
Quick verdict
Agents reproduce NeurIPS-2025 claims at 41/27/15% by artifact tier in this well-audited 100-paper benchmark; one-run cells and contamination cloud the rates.
Summary
RECLAIM is a benchmark of 100 NeurIPS 2025 papers on which AI agents attempt to reproduce each paper's central empirical claim using only the paper and whatever code, data, and weights its authors released. Every task is fixed in advance: a pinned match target, a success tolerance, and a GPU-hour budget. A three-stage pipeline narrows a 1,000-paper sample to 100 test papers (plus 14 for development), stratified into a Run tier (code, data, weights released), a Retrain tier (no weights), and a Reimplement tier (no code), with every availability call and compute estimate human-audited. Four agents — DeepSeek-V4-Flash, Qwen3.6-27B, MiniMax-M2.7, and Muse Spark 1.2 — each run once per paper in an identical harness, and a separate Claude Sonnet 5 auditor grades execution evidence under a frozen rubric with anti-cheat flags, agreeing with three blind human graders at F1 0.82 on the reproduced verdict. The best agent reproduces 41% of Run-tier, 27% of Retrain-tier, and 15% of Reimplement-tier papers (73 of 400 runs overall); failed runs spend on average only 29% of their budget, so compute does not bind. The most common failure, in 63 of 400 runs, is building the method without checking any part against the paper's numbers. Limitations include one run per cell with no confidence intervals, unmeasured training-data contamination, an agent-writable evidence directory without integrity checks, and a single venue and year.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Agent reproduction rates collapse as released artifacts shrink: the best agent reproduces 41% of Run-tier, 27% of Retrain-tier, and 15% of Reimplement-tier papers.
- Compute is not the binding constraint: failed runs use on average 29% of their GPU-hour budget, and 96-hour-band cells spend 6.2% of their grant with 1 of 48 reproducing.
- The most common failure mode (63 of 400 runs) is writing the method without checking any component against the paper's numbers.
- An adversarial LLM auditor with a frozen rubric grades execution evidence reliably enough for scale, agreeing with blind human graders at F1 0.82 on the reproduced verdict.
- Failure modes split into those that recede with model strength (unchecked reimplementation 43→20, build failures 19→5) and those that do not (wrong artifact measured 17→21).
Strengths
Tier design isolates artifact availability, the variable prior benchmarks fixed
Run/Retrain/Reimplement tiers capture field-realistic release states; equal 33/33/33 cells with published venue proportions (33/44/13%, Figure 3) let readers reweight, unlike CORE-Bench which excludes papers without runnable code by construction (Section 2).
Provenance-audited grading with code-enforced integrity
A frozen rubric (Table 7) plus deterministic verdict derivation (Figure 16) forces score 0 on any high-severity flag and caps no-execution runs at unverifiable; the auditor re-executes scoring scripts rather than trusting agent reports (Section 4).
Human-validated at full scale
Actionable failure taxonomy
Exceptional release and refreshability
Weaknesses
Single-run design without uncertainty quantification
Unmeasured contamination on exactly the tiers where agents do best
Evidence directory is agent-writable with no integrity checks
Auditor validation covers only the reproduced verdict
Host environment confounds build-failure conclusions
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers? — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the RECLAIM benchmark?
RECLAIM is a benchmark of 100 NeurIPS 2025 papers on which AI coding agents try to reproduce each paper's central empirical result using only the paper and whatever code, data, and model weights its authors publicly released. Each task has a pre-registered target number, a success tolerance, and a fixed GPU-hour budget, and an AI auditor grades the attempt from execution logs rather than the agent's own report. The set is designed to be rebuilt each year from new conference papers.
How well can AI agents reproduce ML papers?
In RECLAIM's results, the best agent reproduced 41% of papers where code, data, and weights were all released (Run tier), 27% where weights were missing (Retrain), and the best rate where code was missing (Reimplement) was 15% (Muse Spark 1.2; the strongest agent, DeepSeek-V4-Flash, got 12% there). Across all four agents and 400 runs, 73 runs (18%) fully reproduced their target.
Why do AI reproduction agents fail?
The single most common error (63 of 400 runs) was implementing the method without ever checking any component against the paper's reported numbers. Other leading causes were build and dependency failures, running the wrong experiment, and measuring the wrong released artifact. Compute was not the problem: failed runs used on average only 29% of their GPU budget.
Is the RECLAIM dataset and code public?
Yes. The authors release the 100-paper benchmark (plus a 14-paper dev split) on Hugging Face, the harness and rubric on GitHub (github.com/mithils3/reclaim), and verbatim transcripts of every graded run at reclaim-traces.vercel.app, so the grading stage is reproducible even though agent rollouts are not turn-for-turn repeatable.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extracted text omits most of Sections 5.2-5.3 prose and parts of Appendices A, E, H, J; some analysis numbers (e.g., early-stopping statistics) could not be verified.
- Figures are available only as embedded text; exact values in Figures 2, 6, and 9-15 were read from surrounding prose and tables.
- Minor accounting ambiguity in run counts (396 started / 372 labeled / 25 ungraded) could not be resolved from the truncated text.
- References are partially truncated; citation accuracy was not audited beyond in-text usage.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.