LabFactory: Building and Evaluating Executable AI Labs
Jinge Wu, Hongjian Zhou, Mingde Zeng, Jiayuan Zhu, Junde Wu, Jiazhen Pan, Lei Clifton, Andrew Liu, et al.
cs.LG
Quick verdict
LabFactory shows AI builders can deliver working scientific solvers, but selected cases, unverified bars, and contamination risks temper the all-clear claim.
Summary
LabFactory is a framework in which an AI builder (claude-opus-5 in the pi coding-agent harness) turns a scientific brief into an executable 'AI lab': a packaged solver combining models, knowledge resources, tools, and a controller behind a fixed entry-script interface. A separate host runs the delivered artifact on held-out inputs - identifiers reminted, fields permuted, references kept outside the solver interface - and scores it under the task protocol. The paper reports 28 selected constructions across seven categories (molecules/proteins, genomics, imaging, physiological signals, knowledge/reasoning, clinical decision support, biomedical text) whose delivered solvers cleared configured reference bars on all 33 subtests; ten contain models fitted during construction, seventeen assemble tools and retrieval around the platform LLM. Median builds took 15.67 hours, 25.8M tokens, and 117 GPU-minutes. Matched raw-LLM comparisons (12 subtests, 9 cases) show modest gains - up to +8.05 pp on MedMCQA - and the raw LLM already cleared 11 of 12 bars. Main limitations, largely acknowledged by the authors: cases are selected, not a success rate; reference bars are configured thresholds, some rescaled, inferred, or of unknown provenance; train/evaluation overlap (e.g., CASF-2016) is unverified; judge-model identity is unrecorded; and each case is a single run with no variance estimates.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An AI agent (claude-opus-5) can carry a scientific brief to a packaged, invocable solver that clears its configured reference bar on all 33 subtests of the 28 reported constructions.
- Separating the builder from a host-side evaluation (fixed entry script, reminted identifiers, hidden references) makes the delivered artifact, not the builder's narrative, the object of evaluation.
- Ten delivered labs contain models fitted during construction and seventeen assemble retrieval, tools, and control logic around a fixed platform LLM, organized by three recurring controller patterns.
- In matched raw-LLM comparisons, construction adds modest gains (best +8.05 pp on MedMCQA), and the raw platform LLM already clears 11 of the 12 matched reference bars.
Strengths
Execution-grounded evaluation protocol
The host runs the delivered artifact through a fixed entry script on held-out inputs with reminted identifiers and permuted fields (§3.5), tying every reported metric to an executable system rather than the builder's self-report - a cleaner evaluation object than MLAgentBench-style submission files.
Unusual transparency about scope and provenance
§4.1 explicitly states the 28 cases are 'selected because they cleared' and are not a success rate; Table 2 notes 1-7 flag rescaled (‡), inferred (§), and leaderboard-sourced (†) reference bars; §6.2 and Table 3 disclose that the raw LLM already clears 11/12 matched bars.
Fully traceable worked example
Component taxonomy with resource accounting
Weaknesses
Survivorship-only reporting
Train-evaluation disjointness unverified
Weak or mismatched reference bars
Limited incremental value over the raw platform LLM
Opaque model-graded metrics
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
LabFactory (Wu et al., Oxford/UCL/TUM; arXiv 2609.28697) argues that what an AI agent leaves behind - not its development narrative - should be the object of study. A builder (claude-opus-5 in the pi coding-agent harness, v0.80.1) receives a task tuple (d, X, Y, R, m), works in a metered build VM, and packages a solver D = (P, W, K, U, π) behind a fixed entry script; a separate host executes the artifact on held-out inputs with reminted identifiers, permuted fields, and references kept outside the solver interface (§3.5). Headline claims: 28 selected constructions across seven categories clear configured reference bars on all 33 subtests; 10 labs contain build-fitted models, 17 assemble tools/retrieval around a platform LLM; three controller patterns recur (§3.3). The authors are explicit in §4.1 that these are demonstrated constructions, not a success rate.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LabFactory?
LabFactory is a framework in which an AI agent (claude-opus-5) builds a task-specific, executable 'AI lab' - a packaged solver combining trained models, knowledge resources, tools, and control logic behind a fixed entry script - and a separate host evaluates the delivered artifact on held-out data under the task's protocol. Code is at github.com/JINGEWU/LabFactory.
What results does the LabFactory paper report?
28 selected constructions across seven scientific task categories (33 subtests) whose delivered solvers cleared their configured reference bars under host-side execution - e.g., protein-protein interaction AUROC 0.9874, MedQA accuracy 0.9678, Poly(A) F1 0.8977. Median construction cost: 15.67 hours, 25.8M tokens, 117 GPU-minutes. The cases were selected for clearing, so this is not a success rate.
Does the built system beat just using the LLM directly?
Mostly not, in the matched comparisons reported: on 11 of 12 accuracy-matched subtests the raw platform LLM already cleared the reference bar; construction gains ranged from -1.00 to +8.05 percentage points (best: MedMCQA). Construction mattered most for prediction tasks where the builder fitted models such as CNNs and boosted-tree ensembles.
Is LabFactory a rigorous benchmark?
It is a framework plus 28 selected demonstrations, not a success-rate benchmark. The authors state cases were chosen because they cleared their bars; several reference bars were rescaled, inferred, or come from mismatched question sets; and train/evaluation overlap was not verified for the fitted-model cases.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 and 5-6 are available only via captions and inline text; visual details of result plots and diagrams could not be independently verified.
- Appendix C per-episode construction traces and resource records are referenced but not included in the extracted text.
- The arXiv identifier (2609.28697, v1 dated 23 Sep 2026) and builder model name (claude-opus-5) are taken as stated; future model renames could make the archive ambiguous.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.