Divergent strategies and convergent outcomes in autonomous materials discovery
cs.AI · cond-mat.mtrl-sci
Quick verdict
16 replicated LLM agents diverge in strategy but converge on one frontier—and one database artifact; rigorous pre-registered study, single model-harness only.
Summary
The paper asks what repeated open-ended campaigns by identical LLM scientific agents reveal beyond single-run task success. Sixteen separately initialized sessions of one model-harness (alias claude-opus-5, Claude Code 2.1.233) each received the same frozen CoRE MOF 2024 database (12,499 metal-organic frameworks), a methane-storage objective, a pinned RASPA/GCMC protocol and a one-week budget; eight also received seven enforceable verification checks. Strategies diverged immediately into four classes (106–5,006 structures screened), and eight agents voluntarily built 2,253 hypothetical structures. Yet outcomes converged: eleven agents reported the same Cu pts framework (ten ranking it first among retained materials), with ten independent high-accuracy values of 198.9–200.1 cm³/cm³, and an independent reference calculation of 2,344 audit-retained structures found its nine best values all among agents' reports. However, fifteen of sixteen named the same audit-excluded entry as champion—an incomplete deposition whose removed [B12Cl12]²⁻ anions create artificial pore volume (207.2 vs ~200 cm³/cm³). Enforced checks raised fresh-run reproduction from 1/8 to 8/8 (Fisher p=0.001) without detectably improving conclusion validity. The core lesson: replication establishes stability to resampling, not independent confirmation. Limitations: single unpinned model-harness, N=16 with low power for most contrasts, post-hoc strategy classes without a second coder, and an incomplete reference census at analysis freeze.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Strategies of replicated identical agents diverge almost immediately (unique first-five-action sequences per agent; reported screening from 106 to 5,006 structures across four post-hoc classes).
- Despite divergence, agents converge on the same materials frontier: 11/16 report the Cu pts framework (10 rank it first; values 198.9–200.1 cm³/cm³) and all nine highest reference point estimates appear in agents' reports.
- Enforced checks raised fresh-run headline reproduction from 1/8 to 8/8 (Fisher p=0.001) but could not improve conclusion validity because 15/16 agents selected the same audit-excluded, chemically incomplete entry.
- Agreement among replicated agents establishes stability to resampling of one model-workflow, not independent confirmation: database-level errors (406 charge-inconsistent entries; 6 capacity artifacts) propagate as common-mode errors no amount of agent replication can remove.
- Eight agents voluntarily converted the search task into design programs, building 2,253 hypothetical structures, showing agent evaluations should also credit emergent research programs.
Strengths
Novel population-level evaluation design
Sixteen replicated sessions scored against a hidden answer key plus an independently calculated reference landscape (2,344 structures) reveal divergence/convergence properties no single trajectory can show — a genuine methodological contribution beyond task-completion benchmarks like CORE-Bench or AUTOMAT.
Unusual pre-registration and apparatus rigor
SHA-256-pinned frozen database verified per workspace; sealed public-hash commit of charters, rubrics and answer key; pre-analysis amendment (3 Sept 2026) filed before any report was read; smoke phase with two agents excluded by pre-registration; Wilson 95% intervals for primary proportions.
The common-mode-error finding is conceptually sharp and well-evidenced
Quantified convergence, not anecdote
Honest apparatus disclosure with sensitivity analyses
Weaknesses
Severely limited statistical power
Single, unpinned model-harness
Post-hoc strategy classes by a single coder
Reference landscape incomplete at freeze
Third-party re-execution is partially gated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper reframes agent evaluation from task completion to population-level behavior. Sixteen separately initialized sessions of one model–harness received the same frozen CoRE MOF 2024 database (N=12,499, SHA-256-pinned, 9,167 coordinate-distinct), a methane deliverable-capacity objective, a pinned RASPA/GCMC protocol and a one-week budget; eight ('checked') got seven enforceable verification checks, eight ('unchecked') the same norms unenforced. Four claims are made: early strategic divergence; convergence on the same materials frontier; enforcement changes verification behavior but not conclusion validity; and a shared common-mode error, with 15/16 agents selecting an audit-excluded incomplete structure. All four are supported within the study's stated scope, and the framing is careful about what replication does and does not establish.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
When sixteen identical LLM agents independently ran the same open-ended MOF materials search, their strategies diverged but their conclusions converged — including on the same mistake: 15 of 16 picked a database entry whose missing counter-ions create artificial pore volume (207 vs ~200 cm³/cm³). The paper's lesson is that repeating one agent shows stability, not independent verification; shared input errors survive any amount of replication.
What AI model did the agents in this study use?
All sixteen sessions used the same frontier model recorded in transcripts only under the alias 'claude-opus-5', accessed through Claude Code 2.1.233 between 29 August and 1 September 2026. The author notes that no finer snapshot identifier existed, so identical weights across the campaign cannot be guaranteed and the study cannot be rerun on a pinned model.
Why didn't the enforced verification checks catch the bad material?
The checks targeted calculation reproducibility — they made all eight checked agents re-derive their headline numbers from saved inputs (vs 1 of 8 unchecked) — but none validated the chemistry of the deposited structure itself. One agent even computed the structure's +8 charge correctly, then balanced it with negative groups absent from the file. The error lived in the shared database, so varying the search strategy or repeating calculations could not fix it.
Is the code and data from this study available?
Yes. The repository github.com/jihankim929/replicate-study contains the benchmark manifest, both charters, audit and controller code, the answer key, analysis scripts, and all sixteen agent repositories with complete transcripts; a Zenodo archive (doi 10.5281/zenodo.22520942) holds the analysis tables and RASPA input/output decks for headline values. MOF structure coordinates are distributed as identifier-plus-hash manifests only, because part of the set derives from the licensed Cambridge Structural Database.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 are described in legends only; visual content (UMAP embedding, Fig. 4a distribution, Fig. 2 timelines) could not be independently verified from text.
- Supplementary sections and tables (S1–S8) are referenced but not embedded; judgments relying on them (warning-sign tally, exclusion table, sensitivity analyses) rest on the main-text summaries.
- Some references are 2026 preprints whose content could not be cross-checked in this review.
- The review assumes the extracted text is faithful; OCR artifacts around superscripts (cm³/cm³, anion notation) were normalized by context.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.