AIBuildAI-2.5: Efficient Autonomous AI Model Development Through LLM-Guided Tree Search
Peijia Qin, Ruiyi Zhang, Qi Cao, Han Guo, Li Zhang, Pengtao Xie
cs.CL · cs.AI
Quick verdict
Claims LLM-guided tree search, scheduling and model routing enable SOTA autonomous model building (73.3% MLE-Bench medals); gains plausible but un-ablated.
Summary
The paper presents AIBuildAI-2.5, an agentic system that automates AI model development by framing it as code search over a tree of candidate training pipelines. It targets three efficiency weaknesses of prior tree-search agents: noisy reward-based node selection when only a few candidates fit the budget, lack of resource-aware scheduling of training jobs, and uniformly expensive LLM calls. It introduces an LLM judge that scores pending candidates on expected improvement, grounding, and feasibility (0–5 scale), an LLM selector that ranks the pool using search state, a hardware-aware scheduler, and a router that assigns Claude Haiku/Sonnet/Opus tiers to agent roles guided by a cross-run routing knowledge base. On MLE-Bench it reports a 73.3% medal rate, ranking first on the public leaderboard, and it beats MLEvolve on all six evaluated AIRS-Bench tasks (e.g., 20.8% lower MAE on QM9-Cv). Routing matches or beats an all-Opus configuration on three MLE-Bench tasks at 56% lower total cost. Main limitations: no ablations isolating the three components, no variance or significance reporting anywhere, and the AIRS-Bench comparison uses a single baseline.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An LLM judge and selector providing a prior over pending candidates outperforms reward-only selection (Monte Carlo/UCT-style) when the number of executable candidates is small.
- AIBuildAI-2.5 ranks first on the MLE-Bench leaderboard with a 73.3% medal rate across the 75 tasks.
- AIBuildAI-2.5 outperforms MLEvolve on all six evaluated AIRS-Bench tasks under equal budgets and hardware.
- Per-role model routing matches or exceeds an all-frontier (all-Opus) configuration on three MLE-Bench tasks at 56% lower total cost.
Strengths
Sharp diagnosis of the sparse-reward regime
Section 4.1 argues that when each evaluation costs hours of training, trees stay tiny and MCTS/UCT-style reward backup loses statistical meaning; the judge/selector design (Section 4.2) is a direct, well-motivated answer rather than an incidental trick.
Strong headline benchmark result
Section 2.2 reports a 73.3% medal rate on all 75 MLE-Bench tasks without task-specific customization, placing first on the public leaderboard ahead of MARS, ML-Master, InternAgent, R&D-Agent, AIDE, AIRA-dojo, MLEvolve and AIBuildAI.
Cost analysis in explicit dollars
Reward-hacking gate
End-to-end automation details
Weaknesses
No component ablations
No variance or significance reporting
Confounded AIRS-Bench comparison
Scheduler benefit asserted, never measured
Router evaluation is thin
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AIBuildAI-2.5 (arXiv 2609.25047, UCSD) is a systems/method paper on autonomous AI model development. It frames model building as code search over a tree of candidate training pipelines and claims three efficiency contributions: (1) LLM-guided tree search — a judge scores pending leaves 0–5 on expected improvement, grounding in the parent's measured results, and feasibility, while a selector ranks the whole pool using tree state and remaining budget, replacing reward-only MCTS/UCT selection that the authors argue is starved of signal when evaluations cost hours (Sections 2.1, 4.1–4.2); (2) a threshold-based resource-aware scheduler (Section 4.4); (3) per-role model routing across Claude Haiku 4.5/Sonnet 4.6/Opus 4.7 backed by a cross-run routing knowledge base (Section 4.5). Headline claims: first place on MLE-Bench at 73.3% medal rate; wins over MLEvolve on all six evaluated AIRS-Bench tasks (QM9-Cv −20.8% MAE, ZINC −29.8%, SICK-NLI +5.1%, Yelp +1.4%, Solar −4.0%, Web traffic −3.6%); routing matches the all-Opus configuration at 56% lower summed cost.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AIBuildAI-2.5's score on MLE-Bench?
The paper reports a 73.3% medal rate on MLE-Bench's 75 Kaggle-style AI development tasks, which the authors state ranked first on the public leaderboard at the time of writing, ahead of systems such as MARS, ML-Master, InternAgent, R&D-Agent, AIDE, and MLEvolve.
How does AIBuildAI-2.5 reduce cost?
Through three mechanisms: an LLM judge and selector that prioritize which candidate pipelines are actually trained, a scheduler that fits training jobs to current GPU/CPU capacity, and a router that assigns cheaper Claude models (Haiku, Sonnet) to easier agent roles while reserving Opus for hard ones. On three MLE-Bench tasks, routing cut total cost by 56% versus running everything on Opus while keeping scores comparable.
Is code for AIBuildAI-2.5 available?
Yes. The paper points to a public GitHub repository (github.com/aibuildai/AI-Build-AI, release tag v2.5-latest). However, reproducing the headline numbers requires proprietary Claude API models and a 24-hour run on a single A100 per task.
What are the main criticisms of the paper?
It reports no ablations separating its three components (tree-search selection, scheduler, router), no repeated runs or statistical significance for any comparison, and its AIRS-Bench win over MLEvolve changes both the search strategy and the LLM model mix at once, so the cause of the improvement is not isolated.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Values plotted inside Figures 2–4 (per-task bars and cost/score points) are not present in the extracted text; per-task numbers beyond those quoted in the prose, such as exact leaderboard margins, could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.