Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
Xuehai Wang, Haowei Qin, Tongxin Liu, Junkai Li, Buqiang Xu, Jintian Zhang, Yijun Chen, Zirui Xue, et al.
cs.CL · cs.AI · cs.IR · cs.LG · cs.MA · cs.SE
Quick verdict
Rubric-first induction lifts research-agent scores (+2.1/+3.0 RCB, +16.8 AstaBench subset); consistent, but no significance tests; not compute-matched.
Summary
AutoSciRub is an evaluation-first framework for autonomous research agents: before execution, it decomposes an underspecified instruction into atomic scientific goals, grounds them in retrieved literature (5-7 papers per task) and task-visible data, and synthesizes an executable rubric of verifiable criteria; after execution, it verifies the artifact criterion-by-criterion and drives up to three rounds of targeted revision with adaptive early stopping. On ResearchClawBench (40 tasks, 10 domains), the framework improves the overall score in all six model-harness configurations: +2.38, +1.87, +1.99 for GPT-5.4, GLM-5.2, MiniMax-M3 under Codex, and +2.14, +3.11, +3.60 for Claude Code, OpenClaw, OpenScience with DeepSeek-V4-Flash. On a 20-task Easy-split subset of AstaBench E2E Discovery, mean gains reach 12.6-19.4 points (average 16.8), with two harnesses raising completed tasks from 18/20 to 20/20. A cumulative ablation isolates revision as the largest contributor (+2.05 of +3.11), and rubric-guided revision beats rubric-free self-refinement by roughly 2.7x (+2.05 vs +0.77). Main limitations: each configuration runs once with no significance tests or variance estimates, comparisons are not compute-matched, the AstaBench subset is small and non-standard, rubric quality is judged by a single LLM judge, and scientific core coverage slightly degrades after grounding (3.35 to 3.07).
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Inducing an evidence-grounded executable rubric before execution and using it for criterion-level verification and revision consistently improves research-agent output quality across backbone LLMs and agent harnesses.
- On ResearchClawBench, AutoSciRub improves all six model-harness configurations, averaging +2.08 points across three backbones under Codex and +2.95 points across three harnesses under DeepSeek-V4-Flash.
- On a fixed 20-task Easy-split subset of AstaBench E2E Discovery, the framework adds an average of 16.8 points while maintaining or increasing completed-task counts (18/20 to 20/20 for two harnesses).
- Rubric-guided revision yields about 2.7x the cumulative gain of rubric-free self-refinement (+2.05 vs +0.77 over three rounds from the same checkpoint-0 reports).
- Rubric induction improves specificity (1.65 to 4.40), actionability (2.00 to 3.83) and evidence verifiability (1.78 to 4.08) but slightly reduces scientific core coverage (3.35 to 3.07).
Strengths
Consistent paired gains across heterogeneous setups
Table 1 shows the overall ResearchClawBench score improves in all six model-harness configurations (+2.38/+1.87/+1.99 for GPT-5.4/GLM-5.2/MiniMax-M3 on Codex; +2.14/+3.11/+3.60 for Claude Code/OpenClaw/OpenScience on DeepSeek-V4-Flash), with 49/60 domain-level paired comparisons positive; Figure 3 shows all three AstaBench configurations improving.
A real control for the 'more revision helps anyway' alternative
Table 3 compares rubric-guided revision (+2.05 cumulative) against rubric-free self-refinement (+0.77) from the same frozen checkpoint-0 reports (SHA-256 verified, Appendix A.3), and Figure 6a shows per-round gains of 1.43/1.89 persist for tasks not yet stopped, indicating the rubric, not extra rewriting, drives most of the gain.
Unusually transparent protocol reporting
Honest negative self-report on rubric quality
Transfer evidence includes completion counts, not just scores
Weaknesses
No inferential statistics anywhere
Compute mismatch unresolved by an imperfect control
Thin, non-standard AstaBench evidence carries the headline number
Self-verification circularity
Cumulative ablation leaves the key cell missing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AutoSciRub (Section 3) is an evaluation-first plugin layer for autonomous research agents: (1) Automatic Rubric Induction turns an underspecified instruction into a task-specific executable rubric via skeleton induction, scientific literature grounding, task-data exploration, and criterion synthesis (Eqs. 4-7); (2) Rubric-Guided Iterative Revision verifies the produced artifact criterion by criterion (Eq. 8) and drives targeted revision (Eq. 9) with at most three rounds and adaptive early stopping. The headline claims are consistent gains on ResearchClawBench (+2.08 average across GPT-5.4/GLM-5.2/MiniMax-M3 on the fixed Codex harness; +2.95 across Claude Code/OpenClaw/OpenScience on fixed DeepSeek-V4-Flash; Table 1) and a +16.8 average on a 20-task AstaBench E2E Discovery Easy-split subset (Figure 3). The framing — rubrics as execution-time specifications rather than post-hoc judges — is well argued and clearly positioned against expert-authored rubrics (Starace et al. 2025) and automatic rubric generation that lacks grounding.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AutoSciRub?
AutoSciRub is a framework that gives autonomous LLM research agents an explicit, executable rubric. Before running a task it derives task-specific scientific goals from the instruction, grounds them in retrieved literature (5-7 papers) and the task's own data, and synthesizes verifiable criteria; after the agent produces a report, it checks each criterion and drives targeted revisions, up to three rounds with adaptive early stopping.
How much does AutoSciRub improve agent performance?
On ResearchClawBench (40 tasks, 10 scientific domains) it improves overall scores in all six tested model-harness combinations, with average gains of 2.08 points across three backbone LLMs (fixed Codex harness) and 2.95 points across three agent harnesses (fixed DeepSeek-V4-Flash backbone). On a 20-task subset of AstaBench End-to-End Discovery it averages +16.8 points, with two harnesses completing 20/20 tasks instead of 18/20.
Is the improvement just from letting the agent revise its report more?
Probably not entirely: the authors include a rubric-free self-refinement control starting from the same initial reports, where three rounds add only +0.77 points versus +2.05 with rubric-guided revision (about 2.7x difference). However, the two conditions are not perfectly matched in compute or workspace setup, so some residual uncertainty remains.
Is code or data available for AutoSciRub?
The paper links code at github.com/zjunlp/AutoSciRub, and both benchmarks (ResearchClawBench and AstaBench) are public. Exact replication still requires access to several commercial LLM APIs (GPT-5.4, GLM-5.2, MiniMax-M3, DeepSeek-V4-Flash, plus judge models) and three agent harnesses, making it expensive to reproduce end to end.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3-6 are extracted as raw number sequences interleaved with body text; value-to-configuration assignment was reconstructed from the narrative text (which restates the key gains, e.g., 19.36/12.61/18.38 for Claude Code/Codex/OpenClaw), not from intact figure layouts.
- The provided extraction ends after Appendix B's rubric-format examples (a complete JSON block with the authors' own 'omitted' placeholders); any sections the PDF may contain beyond that point were not part of the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.