UltraBench 2: Towards Robust Evaluation of Vision Foundation Models on Ultrasound
Ashwath Radhachandran, Adam Tupper, Christian Gagné, William Speier
cs.CV · cs.LG
Quick verdict
Presents UltraBench 2, a reproducible 21-task ultrasound FM benchmark with rigorous statistics; caveats: public test data, one fixed training recipe.
Summary
The paper introduces UltraBench 2, a standardized benchmark for evaluating vision foundation models on ultrasound image analysis. It covers 11 classification and 10 segmentation tasks from 12 public datasets spanning 10 anatomical regions, with fixed train/test splits, preprocessed 256×256 letterboxed test sets, a unified Hugging Face Datasets interface, and framework-independent evaluation scripts that score CSV/RLE prediction files. The authors evaluate sixteen model variants — twelve ultrasound-specific checkpoints across ten model families plus four DINOv3 references — under full fine-tuning and head-only training, using macro F1 and positive Dice aggregated into four category scores, with error bars combining five-seed variability and 2000-replication bootstrap resampling. Key findings: pathological tasks (tumor segmentation, malignancy classification) are consistently harder than anatomical or non-pathological ones; DINOv3 ConvNeXt-L tops pathological segmentation (0.794), drawing level with ultrasound-specific models, while domain-specific pretraining retains a classification lead (URFM-L, 0.645 pathological macro F1); full fine-tuning dominates head-only training; and several newer models fail to beat older ones. Pretraining–evaluation overlap is documented per model (Appendix H) and contaminated results are flagged. Limitations: reliance on public test data with no hidden holdout, a single hyperparameter recipe not tuned per model, and three models with undisclosed pretraining corpora that cannot be checked for overlap.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- UltraBench 2 is the only benchmark integrating dataset downloading, standardized preprocessing and evaluation, diverse ultrasound task coverage, permissive licensing, and a leaderboard (Table 1).
- Pathological tasks — both segmentation and classification — are consistently harder than anatomical or non-pathological tasks across all evaluated models.
- General-purpose DINOv3 (especially ConvNeXt-L, 0.794±0.008 pathological Dice) has drawn level with ultrasound-specific models on segmentation, while ultrasound-specific pretraining retains a classification lead (URFM-L, 0.645±0.021 pathological macro F1).
- Full fine-tuning consistently matches or beats head-only training, and parameter count alone does not predict performance.
- Several newer ultrasound foundation models (USF-MAE, EchoCare, TinyUSFM) fail to outperform 2023-2025 predecessors under a standardized protocol.
Strengths
Statistical design above subfield norm
Category-level SEs (Eq. 2) combine run variance over S=5 seeds with a 2000-replication bootstrap, resampled at patient level and computed at category level so tasks sharing a test set stay correlated; §3.4 explicitly states SEs are conditional on the fixed split.
Contamination transparency
Appendix H (Table 11) records per-model pretraining overlap; Figures 7-10 bold uncontaminated models; §6.1 flags URFM's CAMUS/Fetal Planes overlap explicitly rather than hiding it.
Careful dataset curation and provenance
Full protocol disclosure
Longevity engineering
Weaknesses
No supervised baseline isolates the 'foundation' claim
Single shared training recipe
Decoder confounded with backbone family
Small-cohort tasks drive unstable per-task rankings
Unauditable overlap for a third of the field
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
UltraBench 2 — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is UltraBench 2?
UltraBench 2 is a standardized benchmark for evaluating vision foundation models on ultrasound images. It includes 21 tasks — 11 classification and 10 segmentation — from 12 public datasets across 10 anatomical regions, published with fixed train/test splits, standardized preprocessing (256×256 letterboxed test images), and framework-independent evaluation scripts. Code, data, and a leaderboard are at github.com/adamtupper/ultrabench2.
What are the main findings of the UltraBench 2 paper?
Three headline findings: (1) pathological tasks (tumor segmentation, malignancy classification) are consistently harder than anatomical or non-pathological ones for every model tested; (2) the general-purpose model DINOv3 ConvNeXt-L tops pathological segmentation with a Dice of 0.794, matching ultrasound-specific models, while ultrasound-specific pretraining still leads on classification (URFM-L, macro F1 0.645); (3) several newer ultrasound foundation models underperform models released in 2023-2025 when evaluated under one common protocol.
Is the code and data for UltraBench 2 publicly available?
Yes. Datasets, models, code, and a public leaderboard are on GitHub (adamtupper/ultrabench2), with most datasets hosted preprocessed on Hugging Face. Two datasets (GBCU and Open Kidney) are gated behind research-use agreements with their original providers and cannot be redistributed; CAMUS and COVID-BLUES carry non-commercial licenses.
How does DINOv3 compare to ultrasound-specific foundation models?
Under this benchmark's fine-tuning protocol, DINOv3 ConvNeXt-L ranks first on pathological segmentation (0.794 Dice) and is statistically tied with the best ultrasound-specific model (SAMUS, 0.826) on anatomical segmentation. On classification, however, the ultrasound-specific URFM-L still leads, especially on pathological classification (0.645 vs 0.583 for the best DINOv3 ViT-L). Caveat: some of SAMUS's and URFM's advantages involve datasets that were in their own pretraining corpora.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-10 were extracted as axis labels and ranking text only; visual details (error-bar overlap, point positions in Figures 4-6) could not be independently verified.
- Review is based on v1 (23 Sep 2026); discussion of trend figures (Appendix D) relies on captions and the extracted axis scales.
- Leaderboard operational details (submission review process, mirror maintenance) are only summarized in the paper text and could not be audited from the manuscript.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.