LabAgent: Customize Any Research Hubs for Scientific Discoveries Using AI Agents
Lei Liu, Yikun Zhang, Jialin Chen, Wanjia Zhao, Rex Ying, Wengong Jin, Hua Xu, James Zou, et al.
cs.AI · q-bio.BM
Quick verdict
LabAgent pairs execution-validated skill libraries with run memory to reproduce published results, leading commercial agents, though evidence is single-run.
Summary
LabAgent is a two-agent framework that converts a laboratory's public artifacts—papers, code, tutorials, issue threads—into an executable skill library, then reuses those skills to reproduce published results. An explore agent searches literature and repositories and admits only skills that pass escalating execution checks; a reproduce agent, built on a shell-command loop, probes the machine, implements, runs, and verifies against published values rather than mere error-free completion, while a persistent memory stores lessons pairing each failure with the exact command that resolved it. Across five life-science settings, LabAgent outscores Claude Code (Opus 4.8) and Codex (GPT-5.5) on every comparison: 32 of 65 TDC ADMET leaderboard entries land inside published standard deviations (mean rank 2.62), BiomniBench-DA mean rubric score is 74.4 versus 72.3 and 65.7, GeneBench-Pro partial credit averages 31.4 with 2 of 10 problems passed, and mean Spearman is 0.443 across 13 ProteinGym assays (7 wins); it also rebuilds a published fine-mapping figure. Limitations, candidly reported: one attempt per target (the skill-library ablation gains +2.8, SE 1.7, not significant), modest absolute accuracy (about half of ADMET entries within the error bar), small public benchmark subsets, higher per-run cost than rivals, and two of five fine-mapping comparators are agent-written surrogates.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A route from a laboratory's public artifacts to executable, execution-validated skills is workable and lands a hand-built agent inside published error bars more often than commercial harnesses (32/65 ADMET entries; mean rank 2.62).
- Experience-conditioned memory that replays exact failure-fix commands lets later reproductions build on earlier ones rather than restart.
- LabAgent outscores Claude Code, Codex and (on ProteinGym) Claude Science on every setting that carries a comparison, across ADMET, BiomniBench-DA, GeneBench-Pro and ProteinGym.
- When no method is named, method choice dominates the outcome (spike assay 0.560 vs 0.021), and the agent can rebuild a published fine-mapping figure with its departures locatable and named.
Strengths
Audit-grade self-candor
The paper names its own failures: the fine-mapping rubric 'cannot see' that two of five lines are surrogates and 'about a third of the total rests on lines that test nothing'; the fastPAINTOR flat power line traces to the agent's own misread enumeration flag (bounded at 0.250/0.200); a README contradicts the call site on coverage (0.80 stated vs 0.95 in effect); and it explicitly refuses run-to-run stability claims.
Verification against the science, not crash-freedom
ADMET scoring compares to leaderboard values ± SD ('a run that completes without crashing but does not recover the published numbers is treated as a failure'); skills enter the library only after a Tier-2 end-to-end execution (Methods 4.1); the ProteinGym sampling rule was fixed before any method ran with nothing removed post hoc ('a sample discarded once its result is known is a sample chosen for its result').
Consistent, costed wins across four comparison settings
Unusually complete reproducibility packaging
Quantified failure economics
Weaknesses
One attempt per target everywhere
Modest absolute performance behind relative wins
Abstract overstates scope of comparisons
Surrogate comparators contaminate the fine-mapping conclusion
Harness-vs-model confound and unexplained fourth arm
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
LabAgent (arXiv 2609.13437; Yale/Northeastern/Stanford/Broad) is a method paper proposing a 'reproduce and discovery harness': an explore agent distills a laboratory's public artifacts — papers, code, tutorials, issue threads — into an execution-validated skill library, and a reproduce agent reuses those skills, conditioned on a persistent failure-to-fix memory, to reproduce published results and extend them. Five life-science settings: TDC ADMET leaderboards, BiomniBench-DA, GeneBench-Pro, ProteinGym variant-effect prediction, and a fine-mapping figure rebuild of the authors' own SuSiE2 study. Central claims: (1) the artifact-to-skill route lands inside published error bars more often than Claude Code (Opus 4.8) or Codex (GPT-5.5); (2) run memory carries fixes forward; (3) first place on every setting that carries a comparison; (4) method choice dominates when no method is named; (5) a published figure can be rebuilt with departures locatable and named.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LabAgent?
LabAgent is a two-agent AI framework described in this paper: an explore agent reads a laboratory's public papers, code and issue discussions and distills them into executable, execution-validated 'skills'; a reproduce agent then runs those skills to reproduce published results, while a persistent memory records each failure together with the exact command that fixed it so later runs can reuse the fix.
How does LabAgent compare with Claude Code or Codex?
Across the paper's five life-science settings, LabAgent scored ahead on every comparison: 32 of 65 TDC ADMET leaderboard entries landed inside published standard deviations (vs fewer for Claude Code and Codex), BiomniBench-DA mean rubric 74.4 vs 72.3 and 65.7, GeneBench-Pro 2/10 problems passed (vs 1 and 0) with partial credit 31.4 vs 28.0 and 16.2, and a mean Spearman of 0.443 on 13 ProteinGym assays with 7 outright wins. Margins are modest and each target was run only once.
Is LabAgent's code and data available?
Yes. The code is MIT-licensed at github.com/fpxlei/LabAgent, all agent prompts are reproduced verbatim in the supplementary information, benchmark sources (TDC ADMET, ProteinGym, BiomniBench-DA, GeneBench-Pro) are public, and the generated skill libraries and reproduction trajectories are provided as Supplementary Data; the UK Biobank genotype panel is access-controlled.
What are the main limitations of the LabAgent paper?
Each target was attempted once, so run-to-run variance is unknown and the one ablation with a stated standard error (+2.8, SE 1.7) is not significant; only about half of ADMET entries landed within the published error bar; public benchmark subsets are small (10 of 129, 13 of 217, 50 of 100); LabAgent cost more per run than rivals; and in the fine-mapping study two of five comparator methods were agent-written surrogates, which the authors themselves flag.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Sampling gaps skip parts of the Discussion, Methods sections 4.5-4.6 and Figure 3's caption; the promised paired t-test values are therefore not visible in the provided text, so headline-gap significance is judged from the stated policy and the ablation's SE.
- Figures appear only as captions and extracted axis labels; plotted values (e.g., Fig. 2a-e, Fig. 5d-i) cannot be checked against the text.
- The 'Claude Science' arm is attested in Table 4 and two text mentions, but its configuration is not described in the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.