Generating Constructive Feedback on Stories via Reinforcement Learning
Maja Stahl, Timon Ziegenbein, Henning Wachsmuth
cs.CL
Quick verdict
GRPO with a priority/specificity/actionability reward makes 7-9B LLMs beat Gemini at story feedback; convincing, but reward-as-metric circularity is a risk.
Summary
The paper presents a reinforcement learning approach for aligning LLMs to generate constructive feedback on creative stories without ground-truth feedback or revision data. Building on the priority/specificity/actionability framework of Rashkin et al. (2025), the authors train three ModernBERT-large regression models as reward functions on 1,920 human-annotated feedback texts from the StoryFeedback corpus (Pearson correlations with human ratings of .537/.865/.651), and optimize Gemma-2-9B, Llama-3.1-8B, and Qwen2.5-7B policies with DrGRPO (QLoRA, k=8) on 908 stories each from StoryFeedback, Storal, and WritingPrompts. On 200 test stories per corpus, GRPO variants substantially raise reward-model scores over instructed baselines (e.g., Gemma priority .668→.930, open prompt, StoryFeedback) and achieve better Bradley-Terry ranks from a three-judge 27B/70B/72B ensemble, outperforming Gemini 2.5 Flash (e.g., rank 2.96 vs 6.11 on WritingPrompts). Ablations show actionability alone yields the best rank (2.15 on Storal) while priority-only ranks worst; human evaluation (5 annotators, 600 instances) confirms the actionability-optimized variant is best on all four dimensions, though inter-annotator agreement is low (Krippendorff's α=.31). Key limitations: the trained reward models double as the primary automatic metric (circularity), reward models are trained and evaluated on the same corpus family, priority prediction is only moderately reliable, and no faithfulness reward component exists.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GRPO with a multi-component reward (priority, specificity, actionability) aligns 7-9B open LLMs to generate more constructive story feedback without any ground-truth feedback or revision data.
- The GRPO-trained models outperform instruction-tuned baselines and Gemini 2.5 Flash across three story corpora in reward-model scores, Bradley-Terry ranks from a three-judge ensemble, and human evaluation.
- Actionability is the primary driver of perceived feedback constructiveness, and an actionability-only reward variant achieves the best rank and human ratings.
- The alignment transfers across story corpora without corpus-specific annotations and is especially effective on original human-authored stories where Gemini underperforms.
Strengths
Ground-truth-free RL formulation with validated reward models
Three ModernBERT-large regressors trained on 1,920 Rashkin et al. (2025) annotations decouple policy training (unlabeled stories) from reward supervision; Table 1 shows pcc .537/.865/.651 for priority/specificity/actionability versus ≤.100 for random/mean baselines, and Appendix E shows reward models track out-of-distribution human ratings better than an LLM-judge ensemble (.43 vs .38 for constructiveness).
Systematic, honest reward ablation
Table 3 runs seven variants across three corpora; the finding that actionability-only achieves the best rank (3.08/2.15/3.10) while priority-only scores highest on priority (.969) but ranks worst on two corpora (5.94 StoryFeedback, 6.48 WritingPrompts) is the paper's most citable result, reported candidly even though it complicates the multi-component framing.
Carefully constructed judge ensemble with bias audit
Cross-corpus and original-story robustness
Strong reproducibility posture
Weaknesses
Circular headline metric
Noisy priority reward without hacking audit
The full multi-component model is not the best system
Narrow human validation and thin baselines
Statistical reporting lacks effect sizes
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This Findings-of-EMNLP-2026 paper (arXiv 2609.04824) aligns three 7-9B instruction-tuned LLMs (Gemma-2-9B-it, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct) via DrGRPO to produce constructive feedback on creative stories, operationalizing three criteria from Rashkin et al. (2025): priority, specificity, actionability. Four headline claims are examined: (1) alignment is possible without ground-truth feedback or revision data, using learned reward models; (2) the result beats instructed baselines and Gemini 2.5 Flash across three corpora; (3) actionability is the main driver of constructiveness; (4) gains transfer across corpora, especially to original human-written stories. All four find support, but claim (2) leans partly on a circular metric, and the ablation contradicts the necessity of the full multi-component reward.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper 'Generating Constructive Feedback on Stories via Reinforcement Learning' do?
It trains 7-9B open LLMs (Gemma-2-9B, Llama-3.1-8B, Qwen2.5-7B) with reinforcement learning (DrGRPO) to write constructive feedback on short stories. Instead of requiring example feedback or student revisions as training targets, it uses three learned reward models that score candidate feedback on priority (does it address the biggest problem?), specificity (is it tailored to this story?), and actionability (would following it improve the story?).
Does the RL-trained model beat Gemini at story feedback?
By the paper's evaluations — reward-model scores and Bradley-Terry rankings from a three-judge LLM ensemble — the GRPO-trained 7-9B models rank above Gemini 2.5 Flash on all three test corpora (e.g., rank 2.96 vs 6.11 on WritingPrompts for Gemma+GRPO with the open prompt). Five human annotators also rated the Gemma-based GRPO feedback above Gemini's (overall constructiveness 4.01 for the actionability variant and 3.69 for the full model vs 3.19 for Gemini on a 5-point scale).
What is the main finding about what makes feedback constructive?
Actionability dominates. An ablation training only on the actionability reward achieved the best judge ranks (e.g., 2.15 on Storal), and human evaluation found the actionability-optimized variant best on all four rated dimensions. Correlation analysis showed actionability (Kendall's τ=.556) and priority (τ=.452) predict overall constructiveness most strongly, while specificity is already high for untrained models.
Is the code and data available, and can the results be reproduced?
Yes. The authors state that experiment code and data are available at github.com/webis-de/EMNLP-26. The three story corpora (StoryFeedback, Storal, WritingPrompts), the base models, and the annotation data from Rashkin et al. (2025) are public, and full hyperparameters, judge prompts, and annotation guidelines appear in the appendices. Full policy training needed 4×H100 GPUs for 16-22 hours per run.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 3's scatter-plot data points are not recoverable from text extraction; assessment of that figure relies on its caption, the surrounding prose, and Appendix D Table 8.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.