EAGER: Enhancing Generative Event Extraction via Reinforcement Learning with Verifiable Rewards
Omar Adjali, Siting Liang, Omair Shahzad Bhatti, Daniel Sonntag
cs.CL
Quick verdict
EAGER's decomposed rewards + schema-contrastive RL lift event extraction +5.15 avg F1 over prior RL, but internal table inconsistencies demand scrutiny.
Summary
EAGER is a post-training reinforcement-learning framework for end-to-end event extraction, applying DAPO with QLoRA to a GoLLIE-7B (Code-LLaMA) backbone. It contributes (1) six decomposed verifiable rewards — format validity, extraction F1 over TI/TC/AI/AC/AI+/AC+, groundedness, over-generation control, coverage, and span precision — combined through Reward-Decoupled Normalization, and (2) Schema-Contrastive Advantage Estimation (SCAE), which samples a different set of negative event schemas for each completion within an optimization group, raising reward variance to counter advantage collapse (validated via Advantage Collapse Rate, Fig. 3). Outputs follow GoLLIE's Python code format augmented with LLM-generated annotation guidelines. Across seven benchmarks (WikiEvents, PHEE, CASIE, GENIA2011/2013, MLEE, M2E2), EAGER reports 30.79 average Mean-F1 versus 25.64 for the strongest prior method, the re-implemented ADELIE-DPO (+5.15), with per-dataset gains of +2.56 to +7.75 (Table 2). Ablations attribute roughly 8 points to SCAE (30.79 vs 22.54) and show argument-level metrics drive the gains (+7.4 vs +2.1 for triggers). Limitations: no significance tests or multi-seed variance; EAGER and EventRL numbers in the main table disagree with appendix per-subtask tables; the GoLLIE '+SFT' rows duplicate the non-SFT numbers; English-only with predefined schemas; a precision-recall tension persists.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Decomposing the RL reward into six verifiable components (validity, extraction F1, groundedness, over-generation, coverage, span precision) gives finer optimization guidance than outcome-level F1 alone.
- Sampling distinct negative-schema sets per completion within each DAPO group (SCAE) alleviates advantage collapse, contributing roughly 8.25 average Mean-F1 (30.79 vs 22.54).
- EAGER outperforms prompting, SFT, and prior RL baselines on all seven benchmarks, with +5.15 average Mean-F1 over the strongest prior method.
- The gains are driven by argument-level metrics (+7.4 avg) rather than trigger-level metrics (+2.1), supporting the task-aligned reward design thesis.
Strengths
Reward decomposition with formal, error-aligned definitions
Table 1 and Appendix F define six rewards (Rfmt, REE, Rgrd, Rovr, Rcov, Rspan), each mapping to a concrete failure mode in the error taxonomy (Table 11): hallucination, extra events/arguments, missing events, boundary errors. Table 7 shows the full reward adds +7.4 on argument metrics vs +2.1 on triggers, demonstrating targeted optimization.
SCAE is a simple, mechanistically validated fix for advantage collapse
Sec. 3.7 varies negative-schema subsets per group member to inject reward variance; Figure 3 shows consistently lower cumulative Advantage Collapse Rate (Eq. 8) with SCAE, and Table 6 isolates a +8.25 avg F1 effect (30.79 vs 22.54) that persists on the stripped REE+Rfmt config (+3.80), ruling out 'richer rewards' as the explanation.
Broad, unified evaluation
Transparent error analysis and training disclosure
Weaknesses
Internal numerical inconsistencies between main and appendix tables
No statistical rigor
Duplicated SFT baseline rows
Unvalidated re-implementations and unspecified prompting protocol
Ablation design and sensitivity reporting are thin
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
EAGER: Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is EAGER in event extraction?
EAGER is a reinforcement-learning post-training framework for generative end-to-end event extraction. It fine-tunes a GoLLIE-7B model with the DAPO algorithm using six verifiable rewards (format validity, extraction F1, groundedness, over-generation control, coverage, span precision) plus a technique called Schema-Contrastive Advantage Estimation that varies negative event schemas across training samples to keep gradients informative. It reports an average F1 of 30.79 across seven benchmarks, above prior RL methods.
How much does EAGER improve over previous event extraction methods?
According to its main table, EAGER reaches 30.79 average Mean-F1 over six subtasks across seven datasets, 5.15 points above the strongest prior method (a re-implemented ADELIE-DPO at 25.64), with per-dataset gains of +2.56 to +7.75. Caveat: the paper's appendix tables list slightly different numbers for EAGER and EventRL than the main table, which reviewers have flagged as an inconsistency.
What is advantage collapse and how does SCAE fix it?
In group-based RL (GRPO/DAPO), all samples in a group share one prompt; with sparse binary rewards they often receive identical rewards, so group reward variance — and hence the policy gradient — is near zero. SCAE (Schema-Contrastive Advantage Estimation) instead samples a different set of negative event schemas for each group member, making discrimination difficulty vary across samples, which raises reward variance and restores informative gradients. The paper measures this with the Advantage Collapse Rate, which stays lower with SCAE throughout training.
Is code and data available for EAGER?
The paper states that code will be released at github.com/OA256864/EE_RL. All seven datasets (WikiEvents, PHEE, CASIE, GENIA2011, GENIA2013, MLEE, M2E2) are public, and hyperparameters are documented (QLoRA rank 8 on GoLLIE-7B, DAPO with 4 samples per prompt, 10 negative schemas, four H100 GPUs for about four hours), making independent reproduction plausible once the code lands.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1, 3, 4, 5, 8-13 are available only as axis labels and captions in the extracted text; curve values cannot be independently verified.
- The main-vs-appendix table discrepancy flagged in this review is computed from the extracted table digits; large and systematic, but should be confirmed against the PDF in case of extraction misalignment.
- Appendix H discusses K and G sensitivity only qualitatively; any supporting tables may not have survived extraction.
- Several references are future-dated within the paper's timeframe (e.g., DAPO as NeurIPS 2026); treated as stated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.