ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation
Yiling Ma, Yilun Zhao, Sihong Wu, Ziyu Chen, Manasi Patwardhan, Arman Cohan
cs.CL
Quick verdict
Post-trains Qwen3-8B on rebuttal-derived review feedback with rubric RL; actionability gains look solid but the judge is GPT-5.4, embedded in its own pipeline.
Summary
The paper formulates actionable peer-review generation as two tasks: given a paper and a target weakness category, generate diagnostic claims (what is wrong) and actionable suggestions (what, where, and how to revise). From OpenReview review–rebuttal threads spanning 15,819 papers (ICLR, NeurIPS, EMNLP), the authors build ActReview-40K: atomic reviewer weaknesses are aligned to rebuttal spans (span F1 0.94 against adjudicated gold), and GPT-5.4 rewrites each pair into initial-review-style claims plus structured suggestions, with the rebuttal serving only as latent supervision. A human-curated benchmark, ActReview-Bench (1,000 instances), supports evaluation. Qwen3-8B-Base undergoes multi-task SFT followed by GRPO with frozen, instance-specific rubrics synthesized offline from references and diverse SFT candidates. ActReview-RL outperforms specialized review-generation baselines (RbtAct, DeepReviewer-14B, OpenReviewer-8B) on judged actionability and grounding while staying near parity with GPT-5.1 and Gemini; ablations attribute gains to rebuttal enhancement (Task 2 ROUGE-L 23.90→35.50) and candidate-aware rubric rewards (Task 2 actionability 65.2→77.6 versus SFT). Robustness is supported by 175 held-out 2025–2026 instances, an independent Claude judge, and 78.3% zero-shot abstention on 360 non-supported pairs. Limitations: technical accuracy only matches frontier LLMs, overstated-necessity errors persist (17.1% versus 5.7% for GPT-5.1schema), and GPT-5.4 pervades data construction, rewards, and judging.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Author rebuttals in real review–rebuttal threads provide effective latent supervision for revision-oriented reviewer feedback (Task 2 ROUGE-L 23.90→35.50 over raw-review training, Table 4).
- Candidate-aware, instance-specific rubric rewards outperform both direct LLM-judge rewards and fixed task-level rubrics in GRPO (Task 2 actionability 77.6 vs 72.4 vs 64.1, Table 5).
- Separating diagnosis from suggestion improves suggestion quality and claim–suggestion alignment versus end-to-end generation (+0.42 and +0.35 GPT-5.4 rubric points, Table 6).
- The resulting 8B model beats prior specialized review generators on actionability and grounding while matching frontier LLMs, and abstains on 78.3% of unsupported weakness–paper pairs zero-shot (Appendix G).
Strengths
Novel, carefully audited supervision source
Rebuttal spans are aligned to atomic weaknesses with P 0.95 / R 0.93 / F1 0.94 against adjudicated gold (Table 1), and the GPT-5.4 enhancement step is audited on 500 instances (κ 0.78, 84.8% acceptability) with explicit leakage and retrospective-rewrite filters.
Multi-layer, partly adversarial evaluation design
Beyond LLM judging: a 200-instance human pairwise eval (93% raw agreement), an independent Claude judge, 175 held-out 2025–2026 instances constructed without rebuttal access (Table 17), and a zero-shot abstention test on 360 non-supported pairs (Appendix G).
Informative ablation ladder for reward design
Transparent failure reporting
Weaknesses
Judge-model coupling pervades the pipeline
Circular references for similarity metrics
Actionability gains coexist with an accuracy ceiling
Incremental delta over closest prior work
Narrow domain and scale
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ActReview targets **actionable peer-review generation**: given a paper and a target weakness label from a 7×17 two-level taxonomy, generate (Task 1) diagnostic claims of concrete deficiencies and (Task 2) revision suggestions specifying what/where/how to revise plus expected outcome (§3.1). The central claim is that author rebuttals in OpenReview threads provide latent supervision for revision-oriented feedback, and that instance-specific rubric rewards during GRPO add diagnostic precision, grounding, and usefulness. The paper ships three artifacts: ActReview-40K (~40K weakness–rebuttal instances from 15,819 ICLR/NeurIPS/EMNLP papers), ActReview-Bench (1,000 human-curated instances), and post-trained Qwen3-8B checkpoints. Headline claims: superiority over specialized review generators (RbtAct, DeepReviewer-14B, OpenReviewer-8B) on actionability/grounding, parity with GPT-5.1/Gemini, and 78.3% zero-shot abstention on unsupported weakness–paper pairs.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ActReview?
ActReview is a post-trained Qwen3-8B model that, given a paper and a chosen weakness category, generates reviewer-style diagnostic claims plus concrete revision suggestions (what to change, where, and how). It was trained on roughly 40K review–rebuttal instances from OpenReview and evaluated on the 1,000-instance ActReview-Bench.
How are author rebuttals used in training?
Rebuttal spans that address a reviewer weakness are aligned offline and used only as latent supervision: GPT-5.4 rewrites each weakness–rebuttal pair into initial-review-style feedback. The model never sees rebuttal text during training or inference, so its output reads like a proactive review comment rather than a response to a rebuttal.
How does ActReview compare to GPT-5.1 or Gemini?
On pairwise judging it clearly beats specialized review models like DeepReviewer-14B and RbtAct, but against GPT-5.1 and Gemini results are near parity overall. Its gains concentrate in actionability, specificity, and revision usefulness; technical accuracy remains comparable rather than superior, and it overstates necessity more often (17.1% vs 5.7% for GPT-5.1schema).
Is the dataset or code available?
The paper states that ActReview-40K and code are released, with links given in Figure 1, and prompts, filters, and audit protocols documented in the appendices. Full reproduction requires GPT-5.4 API access for data enhancement, rubric construction, and reward scoring, plus about four H200 GPUs for training.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Sampled excerpt: middle sections were skipped ('[...sampling gap...]'), including most rows of the main automatic-results Table 2 and parts of Appendices A, C, E, and G; Table 2-dependent judgments here rely on the §6.3 narrative instead.
- The provided text ends mid-way through a figure/example near Appendix G.5; content after that point was not visible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.