IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications
Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan
cs.CL
Quick verdict
Claims LLMs find spec gaps poorly (9.6% DRR) but clarify well once directed (80.6%); evidence-grounded benchmark, though LLM-judged scoring needs validation.
Summary
The paper introduces IdeaAMBIG, a benchmark of 660 evidence-grounded instances testing whether LLMs can judge when a research-idea specification is ready to implement, locate its single method-defining gap, and generate a targeted clarification. Instances pair an underspecified (NOT_READY) specification with a resolved counterpart: 163 real-world gaps mined from reproducibility reports (MLRC, ECIR, TMLR) and GitHub issues, and 497 synthetic gaps created by altering exactly one implementation-critical detail in codification-ready references derived from successful reproductions and AI-Researcher execution trajectories. Human verification shows high agreement (readiness κ 0.84/0.92). Across 13 LLMs, the best model (GPT-5.6-Sol) reaches only 9.6% Macro Defect Recovery Rate on real-world instances but 80.6% clarification-action success when given the annotated defect; an information-bottleneck analysis (13.6 vs 80.6 Macro-CAS) identifies defect discovery, not clarification formulation, as the bottleneck. Oracle studies raise the codification-ready rate from 14% to 98% (n=50) and executable test-pass rate from 45% to 85% (n=20). Limitations: core localization metrics rely on an LLM semantic judge whose human agreement is not shown in the provided text, the real-world subset is small and CV/NLP-centric, and pretraining contamination of evaluated models is not discussed in the visible material.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- IdeaAMBIG is the first benchmark to jointly evaluate codification readiness, method-defect localization, and targeted clarification while separating blocker discovery from the response once the blocker is known (§2).
- Across 13 LLMs, defect localization is the bottleneck: the strongest model (GPT-5.6-Sol) reaches 9.6% Macro DRR on real-world instances but 80.6% clarification-action success given the annotated defect (Abstract, D.2).
- Supplying gold resolutions raises the codification-ready rate from 14% to 98% (Table 2, n=50) and executable test-pass rate from 45% to 85% (D.4, n=20), quantifying the downstream value of successful clarification.
- Human blind review and independent reannotation show both subsets largely contain resolvable, implementation-critical gaps with reliable labels (readiness κ 0.84/0.92; §3.4, Appendix E).
Strengths
Novel problem formalization with a clean three-task decomposition
Codification readiness (§3.1) is framed as the missing link between ideation and execution; separating readiness assessment, defect localization (no hint), and clarification generation (given the defect) lets the paper isolate blocker discovery from action — the design behind its sharpest result.
Evidence-grounded construction with strong human verification
Three triangulated sources (174 of 396 reproducibility reports retained; 121 instances from 4,106 GitHub issues across 50 repos; 139 from 43 AI-Researcher trajectories), two-stage human review with pre-adjudication κ=0.84/0.92 for readiness and 0.92/0.85 for Level-2 labels, plus a target-uniqueness audit reporting 94.5% unique primary blockers (§3.2, Appendix E.2).
Diagnostic depth beyond a leaderboard
Statistical hygiene in the oracle study
Weaknesses
Primary metrics depend on an LLM judge without reported human validation
Synthetic gaps may be cue-detectable
Small, domain-skewed real-world subset and small downstream studies
No decontamination analysis in the visible text
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
IdeaAMBIG targets a stage most research-automation evaluations skip: the specification connecting an accepted research idea to its implementation. The paper defines codification readiness — whether a competent implementer can build a faithful prototype without unsupported assumptions about the core method — and instantiates it as 660 single-defect instances: 163 real-world gaps (42 from reproducibility reports, 121 from GitHub issues) and 497 controlled synthetic gaps. Three tasks are evaluated: readiness assessment (Task 1), defect localization from the specification alone (Task 2), and clarification-action generation given the annotated defect (Task 3). Central claims: (1) across 13 LLMs the best model (GPT-5.6-Sol) reaches only 9.6% Macro Defect Recovery Rate on real-world localization while achieving 80.6% clarification success once the defect is given; (2) localization, not clarification formulation, is the bottleneck (Appendix D.2: Macro-CAS 13.6 end-to-end vs 80.6 defect-guided); (3) oracle resolution lifts the codification-ready rate from 14% to 98% (Table 2, n=50) and executable test-pass rate from 45% to 85% (Appendix D.4, n=20). Claims are correctly scoped in the text and match the visible evidence.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the IdeaAMBIG benchmark?
A benchmark of 660 evidence-grounded instances testing whether LLMs can tell when a research-idea's method specification is ready to implement, locate the single implementation-critical gap, and generate a targeted clarification action. Each instance pairs an underspecified (NOT_READY) specification with a resolved (READY) counterpart, drawn from reproducibility reports, GitHub issues, and executed research projects.
What is the paper's main finding?
Defect localization is the bottleneck: the best of 13 evaluated LLMs (GPT-5.6-Sol) recovers only 9.6% of real-world defects end-to-end (Macro Defect Recovery Rate), but succeeds 80.6% of the time at generating clarifications once told which defect to address. Supplying gold resolutions raises the codification-ready rate from 14% to 98%.
Where does IdeaAMBIG's data come from?
163 real gaps come from MLRC/ECIR/TMLR reproducibility reports and closed GitHub issues on paper-linked repositories; 497 synthetic gaps are made by altering exactly one implementation-critical detail in references verified as successfully reproduced or executed. Human annotators verified atomicity, evidence support, and resolutions (readiness κ up to 0.92).
Is code or data available?
The paper links a code/data release under the name IdeaAMBIG, and its appendices provide task prompts, annotation guidelines, evaluation metric definitions, and the model/decoding configuration used across the 13 evaluated LLMs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The excerpt's sampling gaps skip the main per-model results (Table 1) and Sections 3.4–4.3; headline numbers (9.6% Macro DRR, 80.6% Macro-CAS) are cited from the abstract and conclusion, which are present in the text.
- Appendices A.2, G.2, G.3, H, the Table 8 taxonomy listing and the Appendix I prompts fall in sampling gaps and were not visible; no judgments about their content are made in this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.