Grounded Adjudication of Variations across Extracted TimeLines (GAVEL): Comparing Clinical Timelines Against Their Case Reports
Jack Cummins, Sayantan Kumar, Ketan Tamirisa, Jeremy C. Weiss
cs.AI
Quick verdict
GAVEL: report-grounded LLM judge for clinical timeline comparison, well validated, but merge gains reuse the judging instrument and judge recall is unmeasured.
Summary
The paper introduces GAVEL, an LLM-judge protocol that compares two clinical timelines extracted from the same case report against the report itself, assigning each discrepancy a type (VALUE, TIMING, A_ONLY, B_ONLY, SHARED_UNSUPPORTED, DUPLICATE), a verdict (A, B, BOTH, NEITHER, UNCLEAR), and the supporting report passage, so neither timeline serves as ground truth. The authors first audit the PubMedBERT event matcher underlying prior evaluations, sampling 100 of 40,259 aligned event pairs across four cosine-distance bands: true-match rates fall from 92% below 0.08 to 60% and 48% immediately around the 0.10 cutoff, and 4% above 0.12. Two judge models (GPT-5.6-sol, DeepSeek V3.2) were validated on 2,738 findings with 89.4% and 88.6% confirmation, blinded second-reviewer agreement of 89.8-96.4%, and 99.7% evidence-span locatability. A Bradley-Terry round robin ranks six LLM extractors and two human annotators (GPT-5 top at 1698; annotators A1 1558 vs A2 1359, non-overlapping intervals), replicated at corpus scale over 126 reports (14,493 findings). Guided merging reduced discrepancies attributed to GPT-5's timeline from 7.63 to 0.85 per report, with merged timelines preferred in 77.0% of comparisons. Key limitations: judge recall is unmeasured, the leaderboard uses only 10 reports, the corpus covers one drug class, and the merge evaluation reuses the same GAVEL instrument and one of its judges.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The PubMedBERT 0.10 alignment cutoff used by prior timeline evaluations sits in a distance band where only 60% and 48% of sampled aligned pairs are true matches, so reference-based scores conflate alignment error with extractor error.
- GAVEL adjudicates event-level differences between two timelines against the source case report without designating a reference, with ~89% of emitted findings confirmed by manual review for both judge models.
- Human annotator timelines are not interchangeable reference standards: A1 and A2 have non-overlapping Bradley-Terry rating intervals (1558 vs 1359) on the same corpus.
- GAVEL-guided merging of report-supported edits into GPT-5's timeline is preferred in 77.0% of report-level comparisons and reduces discrepancies attributed to the base timeline from 7.63 to 0.85 per report, as scored by DeepSeek V3.2.
Strengths
Direct audit of the alignment assumption underlying prior metrics
Stratified manual review of 100 of 40,259 PubMedBERT-aligned pairs across four prespecified distance bands shows true-match rates of 92%, 60%, 48%, and 4% (Figure 2, Wilson CIs), locating ~11% of all aligned pairs in the ambiguous 0.08-0.12 region around the operational 0.10 cutoff — a concrete quantification of how reference-based scores conflate alignment error with extractor quality.
Two-judge validation with evidence traceability
2,738 findings from GPT-5.6-sol (89.4% confirmed, 1,376/1,540) and DeepSeek V3.2 (88.6%, 1,061/1,198); blinded second review of 150 findings (agreement 89.8%/96.4%, κ 0.59/0.78, AC1 0.87/0.96); 99.7% of 14,213 quoted evidence spans locatable after normalization, 89.7% byte-exact.
Reference-free design empirically exposes annotator variance
Candid error taxonomy and robustness checks
Honest limitations section
Weaknesses
Stage-4 merge evaluation is self-referential
Judge recall never measured
Leaderboard rests on 10 reports with overlapping intervals
Unjustified scoring degrees of freedom
Judge asymmetry via quantized local model
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
GAVEL targets two structural weaknesses in evaluating clinical timelines extracted from case reports: dependence on a designated reference annotation, and alignment error at the PubMedBERT 0.10 cosine cutoff. The claimed contributions are: (1) an audit showing the cutoff sits in a band where true-match rates collapse — 92% below 0.08, 60% at 0.08-0.10, 48% at 0.10-0.12, 4% at 0.12-0.30 (Figure 2), with ~11% of 40,259 aligned pairs in the ambiguous 0.08-0.12 region; (2) a report-grounded judge protocol (ground → match → compare → adjudicate) emitting six discrepancy types, five verdicts, and verbatim evidence, with ~89% of emitted findings manually confirmed; (3) evidence that human references are non-interchangeable (A1 1558 vs A2 1359, non-overlapping bootstrap intervals); (4) an end-to-end merge experiment cutting discrepancies attributed to GPT-5's timeline from 7.63 to 0.85 per report with 77.0% preference (95% CI 69.8-84.1%).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is GAVEL?
GAVEL (Grounded Adjudication of Variations across Extracted TimeLines) is an LLM-judge protocol that compares two clinical timelines extracted from the same case report against the report text itself. For each difference it returns a discrepancy type (e.g., wrong value, wrong timing, one-sided event, shared-unsupported, duplicate), a verdict stating whether the report supports timeline A, B, both, neither, or is unclear, and the report passage used as evidence — without treating either timeline as ground truth.
Did GAVEL show that LLMs beat human annotators at timeline extraction?
In a 10-report round robin judged by GAVEL, GPT-5 ranked highest (Bradley-Terry rating 1698), ahead of Claude Opus 5 (1672) and o4-mini (1610), with human annotator A1 at 1558 and the top open-weight model GLM-5.2 at 1494. Notably, the two human annotators differed significantly (A2 at 1359), showing human reference timelines themselves vary. These are LLM-judge rankings with overlapping confidence intervals at the top, not independent clinical review.
What problem with existing timeline evaluation does GAVEL address?
Prior evaluations designate one clinician-authored annotation as ground truth and align events using a PubMedBERT cosine-similarity threshold of 0.10. The paper's audit of 100 sampled pairs from 40,259 alignments shows that immediately around that cutoff only 48-60% of aligned pairs actually describe the same clinical event, so aggregate scores mix alignment errors and reference-annotation errors into extractor scores.
How accurate are GAVEL's judgments?
Manual review confirmed 89.4% of 1,540 findings from GPT-5.6-sol and 88.6% of 1,198 findings from DeepSeek V3.2 (2,738 total), with blinded second-reviewer agreement of 89.8-96.4%, and 99.7% of quoted evidence passages locatable in the source reports. Important caveat: this measures correctness among reported findings only — how many discrepancies the judges fail to emit (recall) was not measured.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures survive only as interleaved caption text in the extraction; all figure-derived numbers cited in this review (band rates 92/60/48/4%, rating axis labels 1169-1698) are printed verbatim in those captions, but no additional figure-only data is recoverable.
- No code or data availability statement appears anywhere in the provided text; codeAvailable=false reflects the absence of a statement, not an explicit refusal.
- The Stage-1 primary reviewer's blinding status (to extractor identity) is not stated in the text; only the second reviewer's blinding is described.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.