ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI
Jaehyun Nam, Jinsung Yoon, Yanzhou Pan, Yubo Wang, Rui Meng, Parthasarathy Ranganathan, Tomas Pfister, Tomas Pfister
cs.AI
Quick verdict
Google's ScientistTwo: 80.4% success improving 107 top-venue papers (+25.2% avg gain), but headline evidence leans on AI reviewers used in its own loop.
Summary
ScientistTwo (Google Cloud AI Research) is a fully autonomous multi-agent framework for problem-driven research: given a scientific problem derived from a published paper, it builds runnable baselines, generates and evolves hypotheses, screens ideas on benchmark subsets before full evaluation, runs automated ablations, drafts a conference-style paper, and iterates through a simulated review-rebuttal loop (ScholarPeer) plus a meta-review agent that can trigger further idea refinement. It was evaluated on 107 problems drawn from ICLR, ICML, and NeurIPS papers using Gemini 3.6 Flash and Claude Code/Opus 4.8. Reported results: 86/107 (80.4%) problems improved with average 25.2% relative gain over human SOTA; manuscripts score 7.5±1.3 under ScholarPeer (91.9% acceptance) and 5.7±0.6 under the held-out Stanford Agentic Reviewer (72.1%), far above prior agents; generated artifacts pass four integrity audits (49/49 score verification, 0/49 specification violations, 0/1,814 hallucinated references). Cost is ~$3,765 and 2–3 days per task. Limitations: ScholarPeer is both optimization target and headline metric (Stanford acceptance drops 73.5%→69.4% between review rounds); the abstract's 'consistently outperform' claim is contradicted by its own KVPR case (trails AdaEvolve/EvoX); aggregate gains lack significance tests or confidence intervals; the human study has only 9 reviewers; framework code release is unconfirmed.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ScientistTwo autonomously improves 86 of 107 top-venue research problems (80.4%) with an average 25.2% relative gain over the human state-of-the-art baseline.
- Its AI-authored manuscripts outscore accepted human papers under automated review: 7.5±1.3 (91.9% acceptance) on ScholarPeer and 5.7±0.6 (72.1%) on the held-out Stanford Agentic Reviewer.
- The closed-loop ablation, review-rebuttal, and meta-review stages causally improve output quality (Tables 5–6).
- Generated papers and codebases pass all four integrity audits: 49/49 score verification, 0/49 specification violations, 0/1,814 hallucinated references, 49/49 method-code alignment.
- Gains compound across sequential self-improvement on incremental BPE tokenization: +10.9%, then +9.6%, then +8.2% (Table 9).
Strengths
Evaluation breadth unusual for this field
107 problems spanning ICLR/ICML/NeurIPS papers, each requiring multi-dataset, multi-metric evaluation with subset-first screening (§3), rather than the single-scalar benchmark optimization the paper correctly criticizes in prior agents (§1).
Verifiability is engineered, not merely claimed
Table 7's four-check integrity audit: 49/49 score verification by re-execution, 0/49 specification violations, 0/1,814 hallucinated references, 49/49 method-code alignment with all three refinement agents (without them: 1/50 reward-hacking case, 19/1,840 bad references, 39/50 method-code mismatches).
The pipeline itself is ablated, and costs are disclosed
Includes self-critical evidence
Weaknesses
Circular reviewer ecosystem
Aggregate gains lack statistical treatment
Abstract overclaims against the paper's own evidence
Weak human validation and thin novelty checking
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ScientistTwo (Google Cloud AI Research + Waterloo; arXiv 2609.19644) is a fully autonomous multi-agent framework that takes a research problem — operationalized as advancing a published ICLR/ICML/NeurIPS paper — and, per the authors 'without human intervention', produces a novel method, a verified executable codebase, and a conference-formatted manuscript. Central claims: (1) 86/107 problems improved (80.4%) with average relative gain 25.2% over human SOTA; (2) manuscripts average 7.5±1.3 under ScholarPeer (91.9% acceptance) and 5.7±0.6 under the held-out Stanford Agentic Reviewer (72.1%) (Table 2), versus ≤3.8 and 0% acceptance for seven prior agents; (3) generated artifacts pass all four integrity audits (Table 7); (4) the ablation, review-rebuttal, and meta-review loops causally improve quality (Tables 5–6); (5) gains compound across self-iteration (Table 9: +10.9%, +9.6%, +8.2% on incremental BPE tokenization).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ScientistTwo?
ScientistTwo is a fully autonomous multi-agent AI system from Google Cloud AI Research. Given a research problem derived from a published paper, it independently builds runnable baselines, generates and refines hypotheses, runs experiments across multiple datasets, performs ablation studies, and writes a full conference-style paper with executable code, using a simulated peer-review loop to polish the manuscript.
How do ScientistTwo's AI-generated papers compare to human papers?
Under the ScholarPeer AI reviewer they average 7.5/10 with a 91.9% acceptance rate; under the held-out Stanford Agentic Reviewer, 5.7/10 and 72.1% acceptance — above every prior autonomous agent tested. A small human study (9 reviewers on 33 papers) found overall parity with accepted human papers, though humans were still rated better on methodological rigor.
How much does ScientistTwo cost and how long does it take?
About $3,765 per task (token plus virtual-machine costs) and roughly 2–3 days per full research cycle, with most time spent on idea refinement and experiment execution.
Can ScientistTwo's results be reproduced?
The generated codebases were re-executed and matched their reported scores in the paper's integrity audit (49/49), and generated papers are publicly released. However, the framework itself runs on proprietary frontier models (Gemini 3.6 Flash; Opus 4.8 via Claude Code), costs roughly $3,800 per run, and the paper text does not explicitly confirm public release of the framework code and prompts.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text is truncated mid-paper (parts of §3, §4.1, and Appendices A.1–A.2 omitted); figure-based claims (Figs 1, 8–10) rest on captions only.
- Complete generated case-study papers (VD-STrans, the KVPR placement paper, OFP-Shadow fragments) are interleaved with the main text; numbers were attributed carefully, but misattribution risk remains.
- Reviewer independence (training data, ecosystem overlap) could not be verified from the text; ScholarPeer, ScientistOne, and PaperOrchestra share author overlap with this paper per its references.
- Some extracted numbers carry garbled direction glyphs (speedup arrows, ≈/→); magnitudes were cross-checked for consistency before citing.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.