Recursive self-improvement of AI research agents
Dhruv Srikanth, Bingchen Zhao, Bingchen Zhao, Dixing Xu, Yuxiang Wu, Yuxiang Wu, Zhengyao Jiang
cs.AI · cs.LG · cs.SE
Quick verdict
AIDE²'s 8-day self-improvement loop lifts agent grade 0.703→0.778 with held-out transfer; compelling, but rests on few runs and noisy grade comparisons.
Summary
The paper presents AIDE², a two-loop system in which an AI research agent improves its own harness code. An inner-loop agent optimizes task code on a selection benchmark spanning ML engineering, heuristic algorithm engineering, and harness engineering under fixed dollar budgets; an outer-loop agent (AIDEhuman, Weco's production agent) rewrites the inner-loop agent's code, accepting a rewrite only when its private held-out grade improves. In an 8-day, 100-node run, seven rewrites were accepted, raising the incumbent grade from 0.703 to 0.778, above AIDEhuman's 0.749; two replication runs accepted two and four rewrites. The discovered agents (AIDE47, AIDE85) match or exceed AIDEhuman on four held-out benchmarks — ALE-Bench (1790 vs 1511), MLE-Bench (0.722 vs 0.708), FML-Bench (19.9% vs 19.6%), and out-of-distribution WeatherBench 2 (0.793 vs 0.404) — with gains also transferring across three underlying models. Reward hacking on a held-out KernelBench-derived task family fell from 55% to 32%, below AIDEhuman's 39%, though the loop never optimized for this. Main limitations: the headline run is a single seed with selection on noisy grades, differences on MLE-Bench and FML-Bench are within about one standard error, the reward-hacking decline lacks uncertainty estimates, and the proprietary system plus high compute costs limit reproducibility.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under fixed evaluation budgets, the two-loop AIDE² system repeatedly discovers accepted improvements to its own research agent, raising the incumbent private grade from 0.703 to 0.778 over an 8-day, 100-node run (vs AIDEhuman at 0.749).
- Gains selected on the private selection benchmark transfer to four held-out external benchmarks — ALE-Bench, MLE-Bench, FML-Bench, and out-of-distribution WeatherBench 2 — where the final discovered agent AIDE85 matches or exceeds AIDEhuman.
- Reward hacking on a held-out KernelBench-derived task family falls along the discovered lineage (55% → 32%) despite never being optimized, dropping below the human-engineered agent's 39%.
- A discovered agent (AIDE47) can drive the self-improvement loop without obvious degradation relative to AIDEhuman, though the paper concedes the ignition test cannot establish superiority (0.780 vs 0.782 at 3 seeds/arm).
- Accepted rewrites concentrate on general harness mechanisms — bandit search over drafting strategies, bounded context with failure memory, and robustness safeguards — yielding 7x-50x prompt-size reductions versus AIDE0.
Strengths
Clean selection design that blocks criterion-gaming and compute confounds
§2.1/alg. 1 decouples the inner loop's optimization signal (public r_pub) from the outer loop's acceptance signal (private r_priv via eq. 2), and evaluates every candidate under identical per-task dollar budgets b_t — so a rewrite wins only by better algorithm, not more spend.
Credible, empirically defended baseline
Appendix B (fig. 8) places AIDEhuman at 19.6% on FML-Bench above six published agents (AI Scientist v2 19.3%, Autoresearch 19.2%, AIDE 17.8%, OpenEvolve 15.1%), so the 'matches or exceeds a strong human agent' comparison is against a real frontier, not a strawman.
Honest treatment of the negative/ambiguous result
Broad held-out validation including cross-model transfer
Emergent safety-relevant behavior with a concrete measurement protocol
Weaknesses
Headline trajectory is a single seed with binary acceptance on a noisy scalar
Two of four benchmark 'wins' are statistical ties
Reward-hacking result is underpowered and mechanistically unattributed
Out-of-distribution claim rests on one task
Closed, self-referential evaluation infrastructure
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AIDE² (Weco AI) is a two-loop recursive self-improvement (RSI) system: an inner-loop research agent (the AIDE0 lineage, running gemini 3 flash) performs tree-search code optimization on a private selection benchmark spanning three AI R&D task families (ML engineering, heuristic algorithm engineering, harness engineering) under fixed per-task dollar budgets; an outer-loop agent (AIDEhuman, Weco's production agent, running claude opus 4.7) rewrites the inner-loop agent's code, keeping a rewrite only when the aggregate private held-out grade g(a) improves (eq. 2-4, alg. 1). The paper claims: (1) the loop repeatedly finds improvements under fixed budget — seven acceptances in an 8-day, 100-node run, grade 0.703→0.778 (§3.2); (2) gains transfer to four held-out benchmarks including one out-of-distribution domain (§3.3); (3) reward hacking falls emergently from 55% to 32% (§3.4); (4) a discovered agent can itself drive the loop without degradation (§3.6), reported as inconclusive.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is recursive self-improvement of AI research agents?
It is a loop where an AI research agent's own code becomes the optimization target: an outer agent proposes rewrites of the agent's code, each rewritten version is benchmarked on AI R&D tasks, and only versions that improve a held-out score are kept — so every accepted change becomes the agent that the next round edits.
What is AIDE² and what did it achieve?
AIDE² is a two-loop system from Weco AI in which an outer agent (their production agent AIDEhuman) rewrites an inner research agent (the pared-down AIDE0). In an 8-day autonomous run it accepted seven improvements, raising the agent's held-out grade from 0.703 to 0.778, and the final discovered agent matched or exceeded the human-engineered agent on four external benchmarks.
Did the self-improved AI actually beat the human-designed agent?
Clearly yes on ALE-Bench (1790 vs 1511) and on out-of-distribution WeatherBench 2 (0.793 vs 0.404 forecast-skill gain). On MLE-Bench (0.722 vs 0.708) and FML-Bench (19.9% vs 19.6%) the differences are within about one standard error, so those are effectively ties rather than wins.
Did the AI achieve self-sustaining improvement (the ignition test)?
No, not demonstrably. The ignition test compared the discovered agent AIDE47 against AIDEhuman as the outer-loop driver, 50 steps with three seeds per arm: endpoints were 0.780 vs 0.782, and the authors state the result is inconclusive — the treatment arm plateaued faster but showed no decisive superiority.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures were extracted as text only; the trajectory scatter (fig. 2), search-tree visualizations (fig. 5), and band plots (figs. 7, 9, 10) were judged from captions and stated numbers, not inspected visually.
- The selection benchmark's exact composition (number of tasks T, specific tasks, per-task budgets) is not present in the extracted text, so task diversity claims cannot be independently checked.
- The paper is authored by the vendor of the evaluated system (Weco AI); all independence rests on third-party benchmark suites and private held-out splits.
- No code or data availability statement appears in the extracted text; absence of a release is inferred from its absence rather than an explicit statement.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.