AutoData: Agentic Search for Pre-training Data Selection
Yan Meng, Dhruv Srikanth, Bingchen Zhao, Zhengyao Jiang, Yuxiang Wu
cs.AI · cs.CL
Quick verdict
LLM-agent search over executable data-selection recipes beats curated-pipeline baselines on val-bpb across scales; CORE gains lack statistical support.
Summary
AutoData frames pre-training data selection as a search problem over executable selection algorithms, extending AIDE-style agentic code optimization from model and training code to data. In the NanoChat speedrun harness, an LLM agent (GPT-5.5, Gemini-3-Pro-Preview, or Claude-Opus-4.7) proposes selection programs over a feature bank (lexical statistics, WebOrganizer topic/format labels, Qwen2.5-0.5B perplexity, and LLM-annotated factual/reasoning counts), executes them on the 553M-document ClimbMix pool to draw a 14.37M-document (2.6%) budget, and receives feedback from ten-minute GPT-2 depth-8 proxy trainings under four seeds. Across 200 search steps per LLM, the discovered recipes — composite scores with random backbones, Gumbel tournaments, and stratified quotas — significantly beat random sampling, DCLM, perplexity filtering, and RegMix on val-bpb at model depths 8-20; at depth 24 (1.3B) they only match the default corpus. CORE improvements are numerically positive at most scales but the paper concedes they are not statistically significant. Ablations show cheap features nearly match the full feature set, while LLM annotations give the best proxy result. Limitations include a single corpus, proxy-objective dependence, high annotation cost, reliance on proprietary frontier LLMs for search, and code that is promised but not yet released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An AIDE-style agent searching executable selection programs discovers recipes that outperform DCLM, perplexity filtering, RegMix, and random sampling on a well-curated pool (val-bpb, depths 8-20).
- Recipes discovered on a 125M depth-8 proxy transfer without re-tuning, achieving the best val-bpb from 125M to 897M with statistical significance.
- The discovered recipes improve downstream CORE at most scales, though the paper itself states these gains are not statistically significant.
- Cheap lexical/perplexity features capture most of the attainable proxy gain, while LLM-annotated factual/reasoning features yield the best single result (Table 3).
- Pre-training data engineering can be treated as an agentic ML problem — searched over program space rather than fixed human-designed pipelines.
Strengths
Novel problem framing with clean execution
First systematic application of the AIDE propose-execute-refine loop to pre-training data selection (§1, §3), searching a program space of scoring/stratification/stochastic rules rather than mixture weights — a genuine extension of autonomous ML from model code to data.
Budget-matched, variance-controlled evaluation
All methods select from the same 553,155,584-document ClimbMix pool under the identical B=14,374,266 budget; search-time evaluation uses four proxy seeds (§4.1) and final comparisons use three seeds with paired t-tests (Table 1), isolating selector effects from training noise.
Cross-scale transfer validated, negatives reported
Search output is interpretable and fully specified
Actionable ablations on feature cost
Weaknesses
Abstract overclaims CORE
Search-evaluation circularity (winner's curse)
Asymmetric baseline adaptation
Thin statistics, no multiple-comparison control
Proprietary, non-deterministic search stack; code not yet released
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: AutoData: Agentic Search for Pre-training Data Selection
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AutoData and what does it do?
AutoData is an LLM-agent system built on the AIDE framework that automatically searches for pre-training data selection algorithms. It proposes executable selection programs over document features, trains a small GPT-2 proxy on the selected subset, and uses validation feedback to refine the search, finding recipes that beat standard curation baselines such as DCLM and RegMix on small-to-mid scale models.
Does AutoData's discovered recipe improve downstream CORE accuracy?
Numerically yes at most model scales (for example 0.2727 vs 0.2609 for random sampling at 1.3B), but the authors themselves state the CORE gains are not statistically significant due to high variance. The statistically significant improvements are on validation bits-per-byte at model depths 8 through 20 (125M-897M).
What data and models does the AutoData paper use?
It selects a ~14.4M-document (~2.6%) subset from NVIDIA's 400B-token ClimbMix corpus (553M documents) in the NanoChat speedrun harness, using GPT-2 proxies from 125M to 1.3B parameters, with features from WebOrganizer classifiers, Qwen2.5-0.5B perplexity, and Gemini-annotated factual/reasoning signals.
Is AutoData's code available?
The paper states code will be released at github.com/WecoAI/AutoData; it is promised rather than confirmed released. The two winning recipes are printed in full in the paper and build on the public NanoChat and AIDE codebases, but re-running the search itself requires proprietary frontier LLMs (GPT-5.5, Gemini-3-Pro-Preview, Claude-Opus-4.7).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2 and 5-6 are extracted as raw axis labels and numbers; visual search-trajectory shapes could not be verified.
- Exact shaded (significance) cells in Table 1 are described in the caption but their per-cell pattern was not fully machine-readable from the text.
- Some appendix prose may be truncated; the compute accounting for the 'overnight search' relies on stated per-run timings rather than an itemized budget.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.