Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation
Yumi Lee, Harim Oh, Hyoryung Kim, Minji Kim, Eunsu Kim, Hyeseong Lee, Junya Fukuoka, Andrey Bychkov, et al.
cs.CV · cs.AI
Quick verdict
Curated 10.5k WSI-report benchmark from a MICCAI challenge; valuable dataset and structured-representation finding, but rankings lack significance testing.
Summary
The paper introduces REG 2025, a benchmark for whole-slide-image (WSI) pathology report generation built on a clinically curated Pan-Asia dataset of 10,494 WSI–report pairs from five institutions in South Korea, India, Japan, Turkey, and Germany, standardized to CAP/WHO terminology by pathology trainees plus board-certified reviewers (15.4% of the initial collection excluded). Through a MICCAI 2025 challenge (389 participants, 24 final teams, two test phases of 1,000 held-out slides each), the authors evaluate 24 systems and dissect the methods of 11 top teams. The composite ranking score (0.15×(ROUGE+BLEU)+0.4×KEY+0.3×EMB) favors structured report representations: the top three teams (ICGI 0.8397, ICL_PathReport 0.8202, IMAGINE Lab 0.8047) all used category-aware pipelines, gaining roughly +0.09 BLEU and +0.13 KEY over tokenization-based peers. Phase 2's German subset yielded higher means with lower variance, which the authors read as cross-regional generalization. Qualitative analysis documents numeric hallucination (a 5% prostate tumor volume predicted at 10–90%) and diagnostic overspecification (ADH overcalled as DCIS). Main limitations: no significance testing or confidence intervals behind team rankings, possible metric circularity toward the organizers' own template, European test data from a single contributing German center, and team source code not publicly released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Category-aware structured report representations are the strongest performance differentiator: the top three teams gained roughly +0.09 BLEU and +0.13 KEY over tokenization-based approaches, with Phase-2 gaps of ~0.14 BLEU and ~0.18 KEY.
- Models trained on the Pan-Asia dataset generalize across regions, with the Phase-2 German subset showing higher mean scores and lower variance than the Pan-Asia subset.
- Current WSI report generators systematically exhibit numeric hallucination (tumor volume inflation) and diagnostic overspecification (e.g., ADH overcalled as DCIS), mirroring known pitfalls of routine pathology practice.
- REG 2025 is, to the authors' knowledge, the first large-scale clinically curated Pan-Asia benchmark jointly evaluating report generation and vision-language understanding in computational pathology.
Strengths
Clinically rigorous dataset curation
§2.1 documents a two-stage exclusion pipeline (technical WSI QC, then diagnostic-certainty review by three board-certified pathologists after trainee screening), explicit CAP protocol versions (e.g., breast v1.2.0.0, prostate needle biopsy v1.1.0.0), WHO 5th-edition terminology harmonization, listed IRB/ethics approvals, and a stated 15.4% attrition from the initial collection.
Generalization built into challenge design
Phase 2 mixes 500 Pan-Asia slides with 500 German slides (Table 2), with patient-level splits and Dockerized evaluation; §3.1 reports the German subset showed higher mean and lower variance than the Pan-Asia subset, a concrete directional test of cross-domain robustness.
Composite metric targets known NLP-metric blind spots
Score demonstrably tracks diagnostic error severity
Actionable method taxonomy with quantified payoff
Weaknesses
No inferential statistics behind any ranking claim
Potential metric circularity toward the organizers' template
Generalization claim rests on one not-fully-independent center
Qualitative error analysis is illustrative, not audited
Reproducibility and reporting gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper delivers three coupled artifacts: (1) a clinically curated Pan-Asia WSI–report dataset of 10,494 pairs spanning seven organs from five institutions (South Korea, India, Japan, Turkey, Germany); (2) the REG 2025 MICCAI challenge benchmark (389 registrants from 40 countries, 24 final teams), with a composite clinical evaluation score; and (3) a cross-team analysis of 11 submitted methods. Its central claims: structured report representations and multimodal grounding separate winners from the field; Pan-Asia-trained models transfer to European slides; and the field's dominant failure modes are numeric hallucination and diagnostic overspecification. The authors position REG 2025 as the first large-scale benchmark jointly evaluating WSI report generation and vision-language understanding.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the REG 2025 benchmark?
REG 2025 is a MICCAI 2025 grand challenge and dataset for evaluating AI systems that generate pathology reports from whole-slide images (WSIs). It comprises about 10,494 curated WSI–report pairs from five institutions in South Korea, India, Japan, Turkey, and Germany, covering seven organs. Twenty-four teams were evaluated on 2,000 blind test slides in two phases using a composite clinical score combining ROUGE, BLEU, a keyword score, and a biomedical embedding score.
Which model or team won the REG 2025 challenge?
Team ICGI ranked first overall with a final score of 0.8397, using H-Optimus-1 tile features, ABMIL slide aggregation, and an n-gram-constrained Transformer decoder. ICL_PathReport (0.8202, a Tree-of-Experts hierarchy over Virchow features) and IMAGINE Lab (0.8047, concept-prompt-guided cross-attention) placed second and third.
Is the REG2025 dataset publicly available?
Yes. The dataset is released under the CC BY-NC-SA 4.0 license via reg2025.grand-challenge.org for non-commercial research; commercial and clinical use are prohibited. The evaluation code is on GitHub (github.com/hrb0/reg), but the participating teams' source code was used only for internal reproducibility checks and is not public.
What are the main failure modes of pathology report generation models found in this study?
Two recurring failure modes: numeric hallucination, where quantitative attributes are inflated (a ground-truth prostate tumor volume of 5% was predicted anywhere from 10% to 90% across teams), and diagnostic overspecification, where models collapse uncertain lesions into definitive diagnoses — for example, atypical ductal hyperplasia (ADH) being overcalled as DCIS.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: [...sampling gap...] marks skipped mid-paper content, including parts of §2.1, §3.3 (baseline comparison details), §5 (discussion), acknowledgements/funding, and portions of Tables S1–S2; judgments touching those areas rely on visible text only.
- Figures 1–4 and S1–S4 are present only as captions in the extraction; figure-level evidence (e.g., score-distribution plots, radar plots) could not be independently verified.
- Funding text is cut mid-sentence at a sampling gap ('NRF gra...'); the full acknowledgement exists in the published paper.
- The structured-vs-tokenization subgroup statistics are taken from the paper's own aggregation; full per-metric tables for all 24 teams were not fully visible in the excerpt.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.