arXiv paper2026-09-18· medical imaging machine learning (evaluation methodology)· benchmark audit / empirical study

Beyond Benchmark Scores: Auditing Medical Vision-Language Models for Chest X-Ray Tuberculosis Screening

cs.CV · cs.LG

Quick verdict

Audits 4 medical VLMs on chest X-ray TB screening; rankings, reliability and thresholds often fail to port across cohorts/prompts; rigorous, honest audit.

Summary

This retrospective audit tests whether conclusions about medical vision-language models survive changes in the evaluation specification. Three medical checkpoints (BioMedCLIP, CheXficient, MedSigLIP) and OpenCLIP are scored on 12,200 chest radiograph records from Montgomery, Shenzhen, TBX11K and VinDr-CXR using five fixed prompt families (244,000 model-image-prompt scores), probing ranking, score reliability, negative spectrum, prevalence and threshold-transport claims. No model leads everywhere: prompt-family changes alter AUROC in 21 of 48 Benjamini-Hochberg-controlled comparisons; replacing healthy with sick non-TB controls cuts AUROC by 0.075-0.306; on VinDr-CXR the medical models score below 0.5 AUROC against pneumonia and lung tumor despite 0.780-0.945 against no-finding controls. Thresholds tuned for 95% sensitivity on TBX11K training retain that constraint in only 4 of 16 transports, sometimes with near-zero specificity. A five-seed ResNet-50 reaches 0.999 AUROC on source validation but 0.629 externally; conservative perceptual-overlap exclusion narrows but does not close the gap. CheXficient's documented VinDr-CXR pretraining exposure is flagged throughout. Limitations include image-level rather than patient-level uncertainty, small named-disease control groups (73 lung tumors, 210 pneumonia), no stated code release, a single supervised architecture, and incomplete checkpoint provenance for two models.

Review complete

Assessment at a glance

medical imaging machine learning (evaluation methodology)

risk flags identified

● high 0

● medium 2

● low 3

Reproducibility readinessmedium

Code & data availablenot stated

Review confidencehigh

AI text signal0/20 passages flagged

Deep research viableCode-reproducibleNeeds GPU

5 strengths · 5 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

No vision-language checkpoint leads across all cohorts and reliability criteria; CheXficient and MedSigLIP swap leads and their AUROC differences are statistically unresolved on Montgomery and TBX11K validation (Section 3.1).
Prompt-family specification changes AUROC in 21 of 48 FDR-controlled paired comparisons, so reporting a checkpoint without exact prompts under-specifies the classifier (Section 3.2, Table 6).
Replacing healthy controls with sick non-TB controls reduces AUROC by 0.075-0.306 for all four models, and on VinDr-CXR medical models fall below 0.5 AUROC against pneumonia and lung tumor (Sections 3.3, Tables 9-11).
Thresholds selected for 95% source sensitivity retain that constraint in only 4 of 16 transports by point estimate, and retention can coincide with specificity near zero (Section 3.5, Tables 14-17).
A five-seed supervised ResNet-50 reaches 0.999 AUROC on TBX11K validation but 0.629 on both external cohorts, and conservative perceptual-overlap exclusion narrows but does not close the gap (Section 3.6, Tables 18-20).

Strengths

Pre-committed sequential analysis blocks prevent target tuning
Section 2.1 fixes cohort rules, prompts, scoring implementation and thresholds before outcome tables are assembled, with an explicit disclaimer that this is not full preregistration; no target cohort selects the primary prompt or checkpoint (Sections 2.4, 2.8).
Multiplicity-aware, design-matched inference
48 paired DeLong tests with Benjamini-Hochberg FDR (Section 2.11; 21 survive), Holm correction over 10,000-permutation p-values for the four negative-spectrum contrasts (all ~0.0004, Table 9), and bootstrap designs that change per block (ordinary vs within-label vs shared-positive resampling), each justified.
Consistent pretraining-exposure qualification
Well-controlled paired contrasts isolate the manipulated factor
Exact artifacts disclosed

Weaknesses

No code or score release stated
Checkpoint provenance gaps bound exact reproduction
Small subgroups underpin the headline sub-0.5 finding
Single supervised architecture for the transfer-gap claim
Overlap screen excludes VinDr-CXR and is unadjudicated

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

The paper's central object is not a model but an evaluation specification E = (m, p, C, N, pi, tau, r) (Section 2.1, Eq. 1): checkpoint and processor, prompt and aggregation, cohort, negative class, prevalence, threshold, criterion. Portability means a stated conclusion survives a declared change to one component. Four questions structure the audit (Q1-Q4): ranking stability (Q1), prompt and negative-spectrum effects (Q2), reliability and threshold transport (Q3), and whether supervised source validation plus overlap control resolves transfer (Q4). Claims are deliberately narrow: conclusions attach to frozen configurations and annotated samples, not clinical utility (Section 2.12). This scoping is the paper's main conceptual contribution and is executed consistently - every result is tied back to which specification component changed, culminating in the Table 21 reporting checklist.

Methodology Assessment

The design is a retrospective benchmark audit on 12,200 image records (Table 1): Montgomery 138, Shenzhen 662, TBX11K train/val 6,600/1,800, VinDr-CXR test 3,000. Four frozen checkpoints - BioMedCLIP, CheXficient, MedSigLIP, OpenCLIP - are scored with a two-class softmax over averaged, renormalized prompt embeddings retaining the learned scale (Eqs. 2-3). All prompt texts appear verbatim (Table 3), a reproducibility practice worth emulating. Analyses run in fixed sequential blocks (Section 2.1), preventing target-specific prompt or threshold selection, though the auth

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 5 strengths & 5 weaknesses, analyzed in detail
5 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is the main finding of this paper?

That a high benchmark AUROC for chest X-ray tuberculosis screening does not transfer when the evaluation changes: model rankings flip between cohorts and criteria, prompt wording changes AUROC in 21 of 48 controlled comparisons, results against healthy controls collapse (AUROC losses of 0.075-0.306) against sick non-TB controls, medical models score below 0.5 AUROC against pneumonia and lung tumor on VinDr-CXR, and thresholds tuned for 95% sensitivity keep that guarantee in only 4 of 16 transfers to new cohorts.

Which models and datasets were evaluated?

Three medical vision-language models (BioMedCLIP, CheXficient, MedSigLIP) plus a general-domain OpenCLIP comparator, scored zero-shot on 12,200 chest radiograph records from four public datasets (Montgomery, Shenzhen, TBX11K, VinDr-CXR) using five fixed prompt families, for a total of 244,000 model-image-prompt scores.

Can CheXficient's VinDr-CXR results be trusted as external validation?

No. CheXficient's training documentation lists VinDr-CXR as a pretraining source, so its VinDr results (e.g., AUROC 0.878 against all non-TB images) are a within-dataset stress test rather than independent external validation. The paper marks this exposure in every relevant table and figure, and still shows the model fails against named diseases (AUROC 0.466 vs pneumonia, 0.372 vs lung tumor).

Is code or data available for this study?

The text lists public data sources (Montgomery, Shenzhen, TBX11K; VinDr-CXR via PhysioNet credentialing) and discloses exact prompts, seeds, library versions and threshold values, but no code repository or data-availability statement appears in the provided text, so independent verification currently requires re-implementing the pipeline.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Figures 1-7 were judged from captions, in-text numbers and tables; plotted intervals and reliability-diagram shapes were not independently verifiable from the extracted text.
No data/code availability section appears in the extracted text; a statement in a supplement or v2 would change the codeAvailable and reproducibility judgments.
The abstract in the supplied metadata is truncated mid-sentence, but the complete abstract appears in the embedded body text.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper