Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers
Hanhua Hong, Yizhi Li, Luu Gia Huy, Jian Yang, Ming Zhou, Chenghua Lin
cs.CL
Quick verdict
NLPCC 2026 task overview: process-traced repro benchmark (150 papers); best system 49.6%; rubric-validation correlation is same-judge, limiting its force.
Summary
This NLPCC 2026 shared-task overview introduces AgentActionBench, a process-oriented benchmark for evaluating LLM agents that reproduce experiments from scientific papers. All agent interaction is routed through an MCP-based Action Recorder offering Read, Write, and Execute tools, producing chronological JSON action logs that serve as auditable evidence for rubric-based scoring. Each paper carries rubric items (criterion, importance, stage) across five stages: Paper Observation, Plan Writing, Code Implementation, Command Execution, and Result Matching. The benchmark spans 150 papers — 120 ML from ICML/ICLR/ACL/NeurIPS and 30 AI4Science from venues such as Nature and Bioinformatics. Three students annotated a 15-paper human subset (>1,000 items, avg 67.75/paper) with rotating review; a Claude-based LLM pipeline using MinerU parsing expanded coverage to >10,000 items (avg 69.92/paper). GPT-4o-mini judges each item against tool-filtered logs; scores are importance-weighted pass rates (Eq. 1). Three teams plus a GPT-5.4/Codex baseline competed: the best system (YNU-HPCC-Task11-AgentRep) scored 49.64% overall and the baseline 24.19%, with Command Execution (≤18.04%) and Result Matching (≤17.24%) weakest — execution is the bottleneck. Human- vs model-rubric scores correlate at Pearson 0.93±0.04 and Spearman 0.88±0.07. Main limitations: a single, weak judge model; score-level-only validation sharing that judge; unreported annotator agreement; and no stated public artifact release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Evaluating reproduction through MCP-recorded action traces with paper-specific rubrics yields auditable, process-oriented assessment where output-oriented evaluation of final repositories is unreliable (§3, Fig. 1-2).
- Current LLM agents remain far from reliable experiment reproduction: the best system scored 49.64% and the GPT-5.4/Codex baseline 24.19%, with Command Execution (≤18.04%) and Result Matching (≤17.24%) the weakest stages (Tables 3-4).
- Model-generated rubrics are reliable enough to scale annotation: scores under generated rubrics correlate with human-rubric scores at Pearson 0.93±0.04 and Spearman 0.88±0.07 on the 15-paper human subset (Table 5).
- All participating systems score higher on AI4Science than on ML papers, which the authors attribute to more standard ML methods or simpler experimental configurations (§5.1).
Strengths
Process tracing directly targets a real failure mode
The MCP Action Recorder (§3.1, Table 1) reduces the environment to Read/Write/Execute and logs every call with arguments, results, and timestamps; Fig. 1 motivates this via fabricated execution records and unexecuted-but-plausible code that output-oriented evaluators miss. Table 2 transparently maps each of the five rubric types to its evidence source.
Validated scale-up loop with honest coverage numbers
§3.2 reports a 15-paper human subset (>1,000 items, avg 67.75/paper, three annotators with rotation review) and expansion to >10,000 items (avg 69.92/paper); Table 5 quantifies the human-model agreement (Pearson 0.93±0.04, Spearman 0.88±0.07) instead of asserting reliability.
Negative results reported without spin
Fully specified scoring protocol
Multi-domain extension beyond ML
Weaknesses
Judge fidelity is never validated
Same-judge, score-level-only validation
No inter-annotator agreement statistics
No variance or significance on rankings
Importance weighting and release unspecified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a shared-task overview (NLPCC 2026 Shared Task 11), not a full research paper: it specifies the AgentActionBench evaluation framework, describes benchmark construction, and reports results for three participating teams plus an in-house GPT-5.4/Codex baseline. Three central claims: (1) process-oriented evaluation via MCP-mediated action traces is more auditable than judging final repositories — Fig. 1 motivates this with fabricated execution records and plausible-but-unexecuted code; (2) current agents remain weak at reproduction, with execution the primary bottleneck; (3) LLM-generated rubrics are reliable enough to scale annotation, per strong score correlations with human rubrics. Within the genre's scope, the claims are mostly supported, but claim (3) rests on a validation design with a shared-judge weakness detailed below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AgentActionBench?
A benchmark introduced with NLPCC 2026 Shared Task 11 for evaluating LLM agents that reproduce experiments from scientific papers. Instead of judging only the final code repository, it records every agent action (file reads, writes, shell commands) through an MCP-based Action Recorder and scores the resulting traces against paper-specific rubrics covering five stages: paper observation, plan writing, code implementation, command execution, and result matching. It covers 150 papers (120 machine learning, 30 AI4Science) with over 10,000 rubric items.
How well did AI systems perform in NLPCC 2026 Shared Task 11?
Weakly in absolute terms. The best team (YNU-HPCC-Task11-AgentRep) scored 49.64% overall, a GPT-5.4/Codex baseline reached 24.19%, and the weakest team 7.03%. All systems did far better at reading papers and writing plans/code than at executing commands (at most 18.04%) and matching reported results — execution is the main bottleneck.
Why did agents score higher on AI4Science papers than on ML papers?
Every participant and the baseline scored better on the 30 AI4Science papers than on the 120 ML papers (for example, 61.25% vs 46.69% for the top team). The authors hypothesize these papers use more standard ML methods or simpler experimental configurations, but the paper offers no detailed analysis confirming this explanation.
How reliable are the automatically generated rubrics?
On the 15-paper human-annotated subset, evaluation scores produced by model-generated rubrics correlate with human-rubric scores at Pearson 0.93 and Spearman 0.88 (95% CIs ±0.04 and ±0.07). Caveats: both score sets were produced by the same judge model (GPT-4o-mini), validation is at the score level rather than per rubric item, and the generated rubrics shift weight toward code implementation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure contents (Figs. 1-4) are not present in the extracted text; judgments about figures rely on captions and surrounding prose only.
- The literal '?' in the §2.1 citation list '[18,?,27,29]' may be a PDF-extraction artifact rather than a defect in the source; flagged accordingly rather than asserted.
- The extracted text runs complete through Appendix A (Fig. 4 log example) with no truncation detected at the far end.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.