PaperGym: Rubric-Centered Evolution for Research-Plan Generation
Yuhan Wang, Zhengxi Lu, Yuchen Yan, Kaitao Song, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, et al.
cs.CL
Quick verdict
关于解耦的评分标准数据（泄露率 3.7%）加上 OPSD+GRPO 训练提升了 Qwen3 研究计划质量的断言；虽然取得了实际收益，但这是基于 LLM 判断的，缺乏显著性测试。
Summary
PaperGym 将 arXiv 论文转化为研究计划生成的训练环境，这是一项没有可验证答案的任务。其核心修复是结构性的：问题仅从论文的研究目标和背景中合成，而评分标准仅从方法和实验设计章节中推导，从而将标准泄露率降至 3.7%，而现有评分数据集中的泄露率为 11.90–34.10%（表 3）。每个实例包含十个涵盖方法论创新和实验设计的二元标准，外加七个通用标准。训练中使用评分标准两次：作为基于评分标准的在线策略自蒸馏（OPSD）教师模型的特权上下文，然后作为 GRPO 的单标准奖励（α=0.7）。在 Qwen3-1.7B/4B/8B 上，两阶段调度优于 SFT、单阶段调度和反向排序调度，在五个基准测试的平均值上增加了 +5.6/+5.0/+4.8；训练后的 8B 模型在 ResearchQA 上得分为 73.48，而 Kimi K2.6 为 73.19，并且在相同的仅 OPSD 配方下，PaperGym 训练的模型在三向 LLM 判断比较中获胜率为 58.1%，而 RubricHub 训练的同行获胜率为 28.2%。主要局限性：所有核心得分均来自 LLM-as-a-judge 协议（DeepSeek-V4-Flash 也生成了评分标准），没有报告显著性测试、置信区间或多随机种子方差，且域内基准测试与训练数据使用同一管道构建。作者在附录 C.1 中坦诚地指出了奖励破解（reward hacking，即回复变长）问题。
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Drawing the question from research goal/background and the criteria from method/experiment sections cuts criterion leakage to 3.7% versus 11.90%–34.10% in existing rubric datasets.
- The two-stage OPSD→GRPO schedule outperforms SFT, either stage alone, and the reversed ordering at all three Qwen3 scales, adding +5.56/+5.04/+4.81 to five-benchmark averages.
- Under an identical OPSD-only recipe, models trained on PaperGym-20k win 58.1% of three-way LLM-judged comparisons versus 28.2% for RubricHub Science-trained models, isolating data quality as the driver.
- The trained Qwen3-8B reaches 73.48 on ResearchQA, above Kimi K2.6 at 73.19, judged by an LLM protocol.
Strengths
Leakage is measured and fixed at the source
Structural decoupling (question from goal+background; criteria from method+experiments) plus documented detection prompts (A.2); Table 3 shows 3.73%/4.71%/4.97% for PaperGym-20k/Innov/Design vs 11.90%–34.10% for HealthBench through ResearchPlanGen-ArXiv. The PaperGym-Design rubric prompt even embeds a 'non-inferable from the task alone' self-check (D.3).
Unusually thorough ablations
Rubric source/dimension/extraction model (Table 4), privileged information for the OPSD teacher (Table 5), reward mixing ratio sweep with 7:3 optimal (Table 6), stage-ordering swap across all three scales (Table 7: +2.04/+0.99/+1.31), plus entropy-dynamics analysis (B.5, Figures 4–5).
Candid negative reporting
Full resource release with verifiable construction
Judge reliability is probed, not assumed
Weaknesses
No statistical uncertainty anywhere
Judge circularity
Self-built in-domain benchmarks with very low baselines
Win-rate headline uses a narrower setup than the abstract implies
Cross-domain generalization under-evidenced
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
PaperGym: Rubric-Centered Evolution for Research-Plan Generation — Structured Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'criterion leakage' in rubric-based training?
It is the fraction of grading criteria that can be inferred from the research question alone. If criteria leak, a model can raise its reward by paraphrasing the question instead of actually planning. PaperGym reduces leakage to 3.7% by sourcing questions from a paper's goal/background and criteria from its method/experiment sections; the paper measures 11.90%–34.10% leakage in prior datasets.
What is PaperGym-20k?
A released corpus of 20,000 arXiv-derived instances (about 50% computer science, 25% physics, 25% economics), each containing a research question, a reference answer, 10 instance-specific binary grading criteria, and 7 general criteria — usable as a reinforcement-learning environment for research-plan generation.
How do the two training stages (OPSD and GRPO) work here?
Stage 1 (OPSD): the model distills from its own teacher conditioned on the rubric as privileged context, giving dense token-level guidance. Stage 2 (GRPO): complete research plans are graded criterion-by-criterion against the rubric, and the per-criterion binary verdicts form the reward (0.7 specialized + 0.3 general).
Is the trained model better than frontier LLMs?
Only narrowly and with caveats: the trained Qwen3-8B scores 73.48 on ResearchQA versus 73.19 for Kimi K2.6 — a 0.29-point margin under LLM judging with no significance testing. On five-benchmark averages the trained 8B (38.55) still trails every frontier model in the paper's Table 1 (48.11–54.53, e.g. GPT-5.1).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded text (axis labels and legends of Figures 2–4 and 6) is garbled and interleaved by PDF extraction; figure-derived quantitative claims rely on captions and body prose, while all table numbers are cleanly extracted.
- The extraction appears complete through Appendix D.3 (p. 34) with no mid-sentence cutoff; the far-end truncation allowance was not needed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.