Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation
Daniel P. Jeong, Charles Q. Li, Hossein Hosseiny, Nitya M. Bhalla, Fatma Uyar Morency, Pradeep Ravikumar, Zachary C. Lipton, Michael Oberst
cs.CL · cs.AI
Quick verdict
Claims reference reporting style can flip RRG model rankings; strong radiologist-validated method, but Findings-only setup and o3 dependence temper generality.
Summary
The paper argues that automated evaluation of chest X-ray report generation (RRG) conflates clinical accuracy with conformity to the reference report's writing style. Guided by a board-certified radiologist, the authors build a four-dimension taxonomy (Organization, Completeness, Granularity, Language) spanning 12 axes of reporting-practice variation, and introduce ReRef: a generator-verifier loop built on OpenAI's o3 that rewrites reference reports along one axis while preserving clinical interpretation. Thirteen board-certified radiologists validated 480 (original, alternative) pairs across four datasets: 92.7% of alternatives showed no realism degradation, and 98.8% were rated Equivalent or Minor Discrepancy. Perturbing references for nine open-source RRG models on MIMIC-CXR, CheXpert Plus, IU X-ray, and ReXGradient-160K, all nine metrics shift substantially; CRIMSON is most stable (mean IQR 0.083), RadGraph-F1 most sensitive (0.741). Rankings change: under RadCliQ-v1 on MIMIC-CXR, Libra drops from first to second against CheXOne once normal findings are condensed; 7 of 9 metrics show pairwise reversals that also change bootstrap-CI significance, with larger effects under compound perturbations. Limitations include the Section Assignment axis interacting with Findings-only evaluation, only 61 usable CheXpert Plus studies, an o3-dependent pipeline, and a headline dataset still under PhysioNet review.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Automated RRG metrics conflate clinical interpretation with conformity to the reference report's reporting practices, with perturbation-induced score shifts up to 1.5-2x each metric's natural variation (Figure 4).
- Reporting-practice perturbations of references can reverse statistically significant model rankings; 7 of 9 metrics show CI-level pairwise reversals (Figures 7, E1), e.g., Libra vs CheXOne under RadCliQ-v1 on MIMIC-CXR.
- ReRef produces alternative references judged by 13 radiologists to be realistic (92.7% no degradation) and clinically equivalent in 98.8% of pairs.
- CRIMSON is the most reference-robust metric tested, while entity-based metrics (RadGraph-F1, RadCliQ-v1) are the most sensitive; compound multi-axis perturbations amplify ranking disruption (Appendix F).
Strengths
Expert-validated perturbation method with transparent consensus protocol
Section 2.3/4.1: 13 board-certified radiologists annotate 480 (original, alternative) pairs with a 2-to-3-to-4 annotator escalation; 92.7% (445/480) show no realism degradation and only 1.2% (6/480) Major Discrepancy, with per-axis breakdowns in Figure 3.
Sensitivity is normalized to make magnitudes interpretable
Figure 4 expresses score shifts across 36 (model, dataset) pairs in units of each metric's cross-configuration SD, so claims like RadGraph-F1 shifts of 1.5-2x its natural variation are checkable rather than anecdotal.
Concrete, mechanistic demonstrations rather than aggregate hand-waving
Unusually complete methodological documentation
Compound-perturbation analysis anticipates the obvious objection
Weaknesses
Section Assignment axis partially measures the evaluation protocol, not style
Generator and Verifier are the same proprietary model (o3)
Uneven and small evaluation subsets
No quantified mitigation baseline
Reproducibility of headline perturbations is limited
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ReRef in this paper?
ReRef is a method that uses an LLM (OpenAI o3) in a generate-then-verify loop to rewrite a radiology reference report into an alternative writing style — for example more or less detailed, structured vs free-text, different terminology — while keeping the clinical meaning identical. This lets researchers test whether evaluation metrics measure actual clinical accuracy or merely how closely a model's writing style matches the reference.
Does the choice of reference report change which AI model ranks best?
Yes, frequently. On MIMIC-CXR with the RadCliQ-v1 metric, Libra ranked first against original references but fell to second after references were condensed to focus on abnormal findings, with CheXOne rising to first. Across all datasets, model pairs, and perturbations, 7 of 9 metrics showed ranking reversals that also changed bootstrap-CI significance, and multi-axis style changes amplified the effect.
Which chest X-ray report evaluation metric was most robust to reporting style?
CRIMSON was the most stable in this study (smallest sensitivity to style perturbations), partly because it ignores normal findings by design. Entity-matching metrics RadGraph-F1 and RadCliQ-v1 were the most sensitive, and GREEN sat in between, in part because it scores normal findings and is thrown off when they are condensed.
Is the MIMIC-CXR-Ext-ReRef dataset publicly available?
The authors release 120 radiologist-validated (original, alternative) MIMIC-CXR report pairs; per the paper the dataset is under review for publication on PhysioNet. The full ReRef code pipeline, prompts, and axis definitions are open-sourced on GitHub, but the broader perturbed reference sets used in experiments are not fully redistributable due to data licenses.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as flattened extracted text and captions; box-plot shapes and some per-axis values could not be visually verified, though key numbers are printed in captions or body text.
- The abstract string is truncated mid-dataset-name; the full dataset name (MIMIC-CXR-Ext-ReRef) was taken from the body text.
- Appendix figure labels (E1, F1-F2) were parsed from flattened numeric strings; minor transcription risk on those percentages.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.