AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era
Yunxiang Mo, Tianshi Zheng, Yisen Gao, Rui Wang, Newt Nguyen Kim Hue Nam, Kelvin Kiu Wai Tam, Jiaxin Bai, Yangqiu Song, et al.
cs.AI · cs.CL
Quick verdict
Claims agentic retrieval exposes ideation headroom static tests miss; rigorous, unusually honest benchmark, but LLM-judge validity caps all conclusions.
Summary
The paper introduces AgentIdeaBench, a benchmark that evaluates LLM scientific ideation under two matched settings: Static, where a model reads curated reference papers, and Active, where it must gather its own evidence through a Semantic Scholar search tool under a 10-call budget. Across 35 models (33 with matched outputs; headline analyses use 28 open-weight models) over 40 scored subfields in five disciplines, three LLM critics score originality, feasibility, clarity, impact, and specificity, with originality argued against date-filtered retrieved prior art. Active evaluation carries 4.4× the between-model variance of Static and separates frontier models that Static compresses. The Active−Static gain is capability-gated (r=+0.69): the weakest models are hurt (Gemma-2 27B −0.82) while the strongest gain (GLM-5.1 +1.21), replicating on five held-out Gemini models (within-family r=0.88). Active scores also scale about twice as fast with knowledge-cutoff year (+1.16 vs +0.54 per year). Per-dimension analysis attributes the gain to grounding (feasibility +1.21, clarity +0.63, specificity +0.58) while measured originality stays flat (−0.14, not significant); replay and recall-only controls attribute roughly 77% of the gain to the agentic retrieval process rather than reference content. An exploratory Scientific World Modeling loop is reported as unconfirmed: it fails Holm correction and loses to a compute-matched best-of-3 baseline. Main limitations: every score is an LLM judgment with thin human validation, critic discrimination is weak in dense-literature fields, and the capability gate falls to r=0.182 (not significant) on the full 63-model roster including frontier systems.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Active (agent-controlled retrieval) evaluation carries 4.4× the between-model variance of Static and re-separates frontier models that Static compresses (top-eight span 0.39 Static vs 1.39 Active).
- The Active−Static gain is capability-gated (r=+0.69 on 28 open-weight models, replicating at within-family r=0.88 on five held-out Gemini models), and Active scales about twice as fast with knowledge-cutoff year (+1.16 vs +0.54 per year).
- Agent-controlled retrieval improves feasibility (+1.21), clarity (+0.63), and specificity (+0.58) while leaving measured originality unchanged (−0.14, not significant).
- About 77% of the Active gain is the agentic process of gathering references rather than the reference content (Active−Replay +0.26, p<10⁻³; Replay−Static +0.08, n.s.).
- Scientific World Modeling is an exploratory, capability-gated and unconfirmed direction: its pooled gain (+0.245) fails Holm correction and loses to a compute-matched best-of-3 baseline on three of four backbones.
Strengths
Matched-conditions design with genuine controls
Static and Active share subfields, output format, and scoring (§3.2); recall-only (Appendix K: Static≈Recall<Active) and replay (Appendix L: Replay−Static +0.08 n.s. vs Active−Replay +0.26, p<10⁻³) decompose the effect into content vs process, showing ~77% is the agentic process.
Literature-verified critic with documented calibration
§3.3 requires originality to be argued against date-filtered retrieved prior art; Appendix D documents seven calibration rounds, named failure modes (hindsight suppression, detail-density gaming), and anchors: CORE-7 award ideas at 7.71 vs best model 6.33, weak controls at 1.80.
Statistical rigor unusual for a benchmark paper
Honest null-result and scoping discipline
Confound exclusion done empirically, not rhetorically
Weaknesses
LLM-judge foundation with thin human validation
Capability gate collapses at full-roster scale
Critic−generator family overlap
Uneven domain validity averaged into headline scores
Active bundle and budget not fully separated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AgentIdeaBench (arXiv 2609.07611) is a benchmark for measuring scientific ideation in LLM agents, built around one experimental variable: who controls literature access. Every subfield runs under a Static track (curated reference papers, single-pass synthesis) and an Active track (subfield name plus a Semantic Scholar SEARCH/FETCH tool under a 10-call budget), with identical output format and scoring. The roster spans 35 models — 33 with matched outputs, headline statistics on 28 open-weight models, five Gemini models held out — over 40 densely scored subfields in five disciplines. Three headline claims: (1) Active carries 4.4× the between-model variance of Static and separates frontier models Static compresses (top-eight span 0.39 Static vs 1.39 Active; 62% vs 12% of top-half pairs distinguishable, Appendix I); (2) the Active−Static gain is capability-gated at r=+0.69 (Figure 3), and Active scales ~2.1× faster with knowledge-cutoff year (+1.16 vs +0.54/yr; track×cutoff interaction +0.62/yr, 95% CI [0.41, 0.94]); (3) the gain lands in grounding — feasibility +1.21, clarity +0.63, specificity +0.58 — with measured originality flat (−0.14, n.s.). A fourth, explicitly exploratory claim concerns Scientific World Modeling (SWM).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AgentIdeaBench?
A benchmark from HKUST (with NVIDIA AI Technology Center) that tests whether large language models can generate novel, testable scientific hypotheses under two matched conditions: Static (reading a curated set of reference papers in one pass) and Active (searching the literature themselves with a Semantic Scholar search/fetch tool under a 10-call budget). Each 80–150-word hypothesis is scored 1–10 on originality, feasibility, clarity, impact and specificity by three LLM critics, and originality must be justified against prior work retrieved at scoring time.
What are the main findings of AgentIdeaBench?
Active exploration separates models far better than static tests (4.4× the between-model variance). The benefit is capability-gated: the strongest open-weight models gain over a point (GLM-5.1 +1.21) while the weakest lose (Gemma-2 27B −0.82), and Active scores rise about twice as fast with model knowledge-cutoff year. The gain comes from better grounding — feasibility, clarity and specificity improve while measured originality stays flat — and about 77% of it comes from the search process itself, not from which papers were found.
Is the AgentIdeaBench code and data available?
Yes. The paper states that code, critic prompts and all scoring data are released via GitHub (HKUST-KnowComp/AgentIdeaBench) under MIT for code and CC BY 4.0 for hypotheses and scores. Abstracts are deliberately not redistributed — reproducers must refetch them from the Semantic Scholar API — and the five held-out Gemini models' generated texts are withheld for licensing reasons.
Does AgentIdeaBench prove that LLMs can produce novel research ideas?
No. The authors are explicit that every score is an LLM judgment ('measured originality' under their critic), that human validation is preliminary (critic–human agreement κ=0.55 on 100 pairs), and that scores should not be used for funding, publication or priority decisions. The expert novelty study they consider the most important validation has not been run, and originality shows no improvement from agentic retrieval in their own results.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels (Figures 1–9) extract as reflowed schematic text with garbled axis labels; all numbers cited in this review were taken from the prose and tables, which extracted cleanly and consistently.
- The abstract block in the task metadata ends without a final period, but the complete abstract appears verbatim in the embedded paper body; no truncation affects any claim made here, and the text ends with a complete final sentence (Appendix O).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.