Performance vs Consistency: Evaluating a Foundation Model in Lung-RADS Screening
Benjamin Renoust, Pierre Baudot, Tiffany Foriel, Yousra Haddou, Charles Voyton, Pierre-Henri Siot, Ezequiel Geremia, Danny Francis, et al.
cs.CV · cs.AI
Quick verdict
A fair MRMC study: fine-tuned MedGemma AUC 0.83 vs radiologists 0.90 on NLST Lung-RADS; NLST-only training and no external validation temper the claims.
Summary
This preprint compares a medical foundation model against radiologists in a structured lung cancer screening setting. Twelve board-certified radiologists performed Lung-RADS v2022 readings on 481 NLST chest CT scans (132 biopsy-confirmed cancers, 27.4% case-enriched prevalence), with each case read independently by three of the twelve readers (1,443 readings total, single time-point, no priors). Two MedGemma configurations were evaluated: the native google/medgemma-1.5-4b-it zero-shot model using coarse-to-fine prompting over 25-slice z-axis convolutions with max aggregation, and a QLoRA fine-tuned version trained on the NLST Train3 split (7,585 patients). Radiologists achieved a mean AUC of 0.90 (range 0.80–0.94) with fair inter-reader agreement (Fleiss' Kappa 0.329, ICC 0.683); native MedGemma reached AUC 0.70 and the fine-tuned model 0.83, the latter trading sensitivity (0.63) for specificity (0.91) and matching the weakest reader when evaluated on that reader's case set. The authors frame deterministic model outputs as a consistency advantage over inter-reader variability, supporting complementary clinical use. Key limitations, largely acknowledged by the authors: training and evaluation both draw on NLST, leaving open possible train/test overlap with the Train3 split and overfitting risk; the cohort is case-enriched; no external validation; no ablation of the prompting design; and determinism was not tested under perturbations.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Radiologists performing Lung-RADS v2022 on the NLST subset achieve a mean AUC of 0.90 (range 0.80–0.94) with only fair inter-reader agreement (Fleiss' Kappa 0.329, ICC 0.683).
- Native MedGemma (AUC 0.70) fails clinically relevant performance, while QLoRA fine-tuning on the NLST Train3 split raises AUC to 0.83, within the lower range of individual radiologists.
- Fine-tuning shifts the operating point toward specificity (0.91 vs 0.58 native) at the cost of sensitivity (0.63), and the fine-tuned model matches the weakest radiologist on that reader's exact case set (AUC 0.84 vs 0.80).
- Deterministic model outputs under fixed conditions constitute a consistency advantage over inter-reader variability, supporting complementary clinical use in settings with limited expertise.
- These conclusions are limited by case-enriched NLST-only evaluation, absence of external validation, and possible dataset overfitting of the fine-tuned model.
Strengths
Genuine multi-reader human reference
12 board-certified radiologists produced 1,443 independent Lung-RADS v2022 readings (§3.2), and Table 1 quantifies agreement three ways (mean L1 0.892; Fleiss' Kappa 0.329; ICC 0.683), stratified junior vs senior (Kappa 0.235 vs 0.379) — model claims are grounded in measured human variability rather than a single consensus reading.
Same-set control for the weakest-reader comparison
Table 2 re-evaluates both model variants on the weakest reader's exact case set (FT 0.84 [0.76–0.92] vs that reader's 0.80 [0.67–0.92]), addressing the case-mix confounding created by per-reader case allocation.
Candid, specific limitations section
Operating-point transparency
Public code and method disclosure
Weaknesses
Unaddressed train/eval provenance risk
Abstract misrepresents the reading design
Non-standard statistical comparison
Consistency claim rests on untested determinism
No ablations for the central pipeline choices
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This preprint (arXiv 2609.22281v1, Sept 2026), an industry-led study by Median Technologies/eyonis, asks three questions (§1): can a general medical foundation model perform malignancy prediction in a Lung-RADS setting, does fine-tuning help, and how do model predictions compare with measured radiologist variability? It benchmarks MedGemma (google/medgemma-1.5-4b-it) in native zero-shot and QLoRA fine-tuned configurations against 12 board-certified radiologists performing Lung-RADS v2022 on an NLST-derived cohort. Headline claims: radiologists reach mean AUC 0.90 (range 0.80–0.94) with fair agreement (Fleiss' Kappa 0.329); native MedGemma scores AUC 0.70, fine-tuned 0.83; deterministic inference offers a consistency advantage over inter-reader variability. The authors are appropriately explicit that this is a first step, not clinical validation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How accurate is MedGemma compared to radiologists for Lung-RADS lung cancer screening?
In this study, radiologists averaged an AUC of 0.90 (range 0.80–0.94), the fine-tuned MedGemma model reached 0.83, and the untuned version 0.70 on the same NLST-derived cohort. Evaluated on the weakest reader's cases, the fine-tuned model scored 0.84 against that reader's 0.80 — so it performs like a lower-tier reader, not an expert.
How much do radiologists disagree on Lung-RADS categories?
The paper reports fair inter-reader agreement: Fleiss' Kappa of 0.329 overall (0.235 for junior readers, 0.379 for senior), an ICC of 0.683, and an average pairwise gap of about 0.89 Lung-RADS categories — meaning two readers typically differ by just under one category step. Senior readers agreed more than juniors.
Are AI models like MedGemma more consistent than radiologists?
Only in a narrow sense: with fixed prompts and temperature zero the model produces identical outputs on identical inputs, which the paper frames as a consistency advantage over human inter-reader variability. The authors themselves note this demonstrates only repeatability — no perturbation or repeated-run tests were performed — so it should not be read as clinical reliability.
What data was used to train and test the model, and could results be biased?
Both come from the NLST trial: fine-tuning used the Train3 split (7,585 patients) from an earlier study, while testing used 481 patients (132 cancers) selected from the authors' separate annotation effort. The paper does not confirm the test patients were excluded from the training split, and the cohort is enriched to 27.4% cancer versus ~4% real screening prevalence — so absolute performance may be optimistic.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 (ROC curves, per-reader metric bars) appear only as captions in the extracted text; per-reader values beyond the min-reader rows of Table 2 could not be verified.
- Fine-tuning and inference details beyond those stated in the paper reside in the external GitHub repository and were not inspected.
- The author affiliation block is un-mapped in the extraction; affiliation assignments (including one duplicated ORCID) could not be fully verified.
- This is an arXiv v1 preprint (September 2026), apparently not yet peer-reviewed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.