Beyond In-Distribution Metrics: A Systematic Out-of-Distribution Evaluation of Congenital Heart Disease Segmentation
Aniketh Vijesh, Shrisharanyan Vasu, Abhijit Ramesh, Clare Pomeroy-Ward, Harikrishnan Anil Maya, Sarin Xavier, Mahesh Kappanayil, Gilad Gressel
cs.CV · cs.AI
Quick verdict
First systematic OOD evaluation of CHD segmentation; solid and honestly reported, but single target cohort, pipeline confounds, and no significance tests limit conclusions.
Summary
The paper presents the first systematic out-of-distribution (OOD) evaluation of deep-learning segmentation for congenital heart disease (CHD). Four architectures (nnU-Net, Zhu-Net, CardiacSeg, SwinUNETR) plus MAE- and JEPA-pretrained SwinUNETR variants are trained on a private multi-vendor CT cohort (3D-Labs, 79 cases) optionally combined with public CMR (HVSMR-2.0, 60 cases), then tested zero-shot on the public ImageCHD CT cohort (110 cases) using Dice, HD95, and ASSD over six cardiac structures, with five seeds per condition. In-distribution ranking reverses under shift: nnU-Net leads on validation Dice (0.77) but collapses to 0.51 on ImageCHD, while SwinUNETR variants reach 0.62-0.68 (scratch: 0.67), and Zhu-Net fails outright (0.15). Removing CMR data hurts nnU-Net most (0.51 to 0.36). Adding only 11 labeled ImageCHD cases lifts all SwinUNETR variants above 0.76 Dice on a fixed 22-case holdout. Limitations: a single OOD target dominated by septal defects, architecture confounded with preprocessing/training pipelines (nnU-Net self-configured; others resized to 128^3), no statistical significance testing, and reliance on a private source cohort.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- In-distribution validation ranking does not predict cross-cohort performance: nnU-Net achieves the highest validation Dice (0.77) but falls to 0.51 on ImageCHD while SwinUNETR reaches 0.67.
- SwinUNETR variants are the strongest evaluated methods on ImageCHD across combined CT+CMR training, CT-only training, and limited target-domain adaptation.
- MAE and JEPA pretraining provide only marginal gains over scratch SwinUNETR in this setting, with their benefit concentrated in the zero-shot regime.
- Joint CT+CMR training improves ImageCHD transfer for most evaluated methods, but the magnitude is strongly method-dependent (nnU-Net drops from 0.51 to 0.36 without CMR).
- Eleven labeled ImageCHD cases suffice for all SwinUNETR variants to exceed 0.76 Dice on a fixed 22-case holdout.
Strengths
Genuine gap filled with a systematic design
The Introduction correctly notes existing CHD segmentation work is evaluated in-distribution; the paper then varies architecture (4 methods + 2 pretrained variants), training cohort (CT+CMR vs CT-only vs adaptation) and seed (5 per condition), evaluated with three complementary metrics (Dice, HD95, ASSD, Sec. 2.2/3.1).
Multi-angle result reporting
Beyond aggregate results (Fig. 2), the paper gives per-structure Dice with SDs (Table 2), per-diagnosis breakdown across 16 CHD groups (Fig. 3), qualitative comparisons (Fig. 4), and a sample-efficiency curve on a fixed 22-case holdout (Fig. 6).
Honest, well-calibrated interpretation
Actionable sample-efficiency finding
Weaknesses
Single OOD target cohort
Architecture confounded with pipeline
Zhu-Net baseline validity unexamined
No inferential statistics
Limited reproducibility
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper on CHD segmentation OOD evaluation?
In-distribution validation performance is a poor predictor of out-of-distribution robustness in congenital heart disease segmentation: nnU-Net scored highest on validation Dice (0.77) but dropped to 0.51 on the held-out ImageCHD CT cohort, while SwinUNETR-based models retained roughly 0.64-0.68 Dice under the same shift.
Which model generalized best to the out-of-distribution ImageCHD dataset?
SwinUNETR and its MAE/JEPA-pretrained variants (about 0.64-0.68 Dice on ImageCHD) clearly outperformed nnU-Net (0.51), CardiacSeg (0.58), and Zhu-Net (0.15), under both combined CT+CMR training and CT-only training.
How many labeled target-domain cases are needed to adapt the models to ImageCHD?
Adding only 11 labeled ImageCHD cases raised all SwinUNETR variants above 0.76 Dice on a fixed 22-case holdout; self-supervised (MAE/JEPA) pretraining helped mainly in the zero-shot setting and its advantage faded once target labels were available.
Is code or data available for this CHD segmentation benchmark?
The paper states no code or model release. Two of the three cohorts (HVSMR-2.0 and ImageCHD) are public, but the primary training cohort (3D-Labs, a private multi-vendor CT set) is not, so full reproduction of the reported setting is not currently possible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-6 are available only as captions and prose in the extracted text; per-model values for Figs. 5-6 come from prose statements and could not be independently checked.
- The extraction appears complete (abstract through references, Table 2 fully present), but any supplementary material or appendix was not included.
- Seed-level variance and per-diagnosis values beyond what Table 2 and prose state could not be verified from text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.