ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs
Sebastián Andrés Cajas Ordóñez, Maximin Lange, Quang Bui, Anqi Peter Li, Felipe Ocampo Osorio, Rafi Al Attrach, Kushul Reddy Palakala, Sahil Kapadia, et al.
cs.CV · cs.AI
Quick verdict
Paired image-swap audit shows VLMs answer from reports not images; rigorous and honest, but report-derived labels limit clinical conclusions.
Summary
The paper introduces ModaLens, a paired image-swap audit that quantifies how report availability changes the image sensitivity of medical vision-language models. Each of 3,199 MIMIC-CXR test cases (293 patients) is presented twice to MedGemma-27B with the same 14 CheXpert-derived yes/no questions and the same report, differing only in whether the image is the case's own or a substitute study's, yielding 44,786 paired trials. Under an explicit answer instruction, the generated answer changes on 4.26% of trials with the report versus 20.94% without, a paired difference of 16.7 points (patient-clustered 95% CI 15.6-17.7); the original prompt with a lowercase first-token readout gives 4.70% versus 17.07%. Continuous answer margins move more without the report (mean absolute change 2.31 vs 0.69 logits), and the direction replicates across MedGemma-4B, Qwen3.5-9B/27B, and LLaVA-NeXT/Mistral-7B. Attention knockout shows blocking report-token access at shallow layers restores image sensitivity. The effect is heterogeneous: two of fourteen findings reverse direction, and the report-absent rate is readout-dependent. The main limitation, stated prominently by the authors, is that all labels derive from reports, so the audit measures sensitivity, not visual correctness; the manipulation is tested on a single dataset. Code, prompts, and run records are public.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Removing the report raises MedGemma-27B's image-swap answer-flip rate from 4.26% to 20.94% under an explicit answer instruction, a paired difference of 16.7 points (95% CI 15.6-17.7).
- Substitutions move continuous answer margins on most trials even where the binary answer does not flip (mean absolute change 2.31 without vs 0.69 with the report; larger without on 82.9% of trials).
- The direction replicates across three model families and two sizes, but what replicates is report anchoring, not image competence - no tested model reads the substituted image above chance on the label-conditioned one-question set.
- Cumulative attention knockout on report tokens restores image sensitivity only when initiated below roughly layer 22 of 62, bounding when direct report-token access matters.
- Because all labels derive from reports, the audit measures sensitivity to the image intervention, not visual correctness.
Strengths
Clean paired causal design with complete accounting
Within-case pairing holds report and question fixed and swaps only the image; Table 1 documents the full cohort (3,199 cases, 293 patients, 44,786 trials, substitute reuse caps, 376 reciprocal pairs), and substitute sampling enforces an SSIM ceiling of 0.9 (observed max 0.78, Appendix B).
Readout and protocol robustness rare for this literature
The lowercase first-token proxy is validated against token-family and generated-answer readouts under an explicit Yes/No instruction on both designs (Tables 3, 5, 19); prompt order is rerun on the full cohort (Table 2: +12.4 image-first vs +7.8 text-first); substitute resampling over three seeds moves the one-question report effect by at most 0.34 points.
Uncommon transparency about heterogeneity and failures
Cross-family replication under the validated protocol
Text-content controls isolate the mechanism
Weaknesses
Report-label ceiling on all conclusions
Headline readout fragility
Single dataset carries the manipulation
Pooled effect masks non-uniform behavior
Magnitudes are protocol-specific
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ModaLens asks whether a medical VLM actually uses the image when a radiology report is in context. The design is a paired counterfactual audit: 3,199 MIMIC-CXR test cases from 293 patients, each rendered twice (own image vs a substitute study's image, usually same-patient and label-differing), with report and question fixed - 44,786 paired trials across 14 CheXpert-template questions (13 findings plus one composite) on MedGemma-27B. The central claim is causal but about sensitivity, not accuracy: removing the report raises the answer-flip rate from 4.26% to 20.94% under an explicit answer instruction (paired 16.7 points, patient-clustered 95% CI [15.6, 17.7], Table 3); the original uninstructed prompt with a lowercase first-token readout gives 4.70% vs 17.07% (12.37 points, Table 2). The authors explicitly disclaim visual-correctness conclusions because every label derives from report text.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does ModaLens measure?
It is a paired audit that swaps the chest X-ray image while keeping the radiology report and question fixed, then counts how often the model's answer changes. This estimates how much the model's output actually depends on the image versus being driven by the report text.
What is the paper's main finding?
MedGemma-27B's answer changes on only 4.26% of trials when the image is swapped and the report is present, versus 20.94% when the report is removed - a paired difference of 16.7 percentage points (95% CI 15.6-17.7). The report therefore strongly suppresses the model's sensitivity to the image.
Does this mean the model is visually wrong when it follows the report?
No. All evaluation labels in the study are derived from reports, not from the images themselves, so the audit cannot determine whether following the report is correct. An accurate report is an expert reading, and deferring to it may be appropriate; independent image annotations would be needed to settle that.
Is the finding specific to one model or dataset?
The direction replicates across three model families (MedGemma-4B/27B, Qwen3.5-9B/27B, LLaVA-NeXT on Mistral-7B), but the manipulation is tested only on MIMIC-CXR chest radiographs from a single institution, and effect sizes depend on prompt order and answer-readout method.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Text extraction truncated several segments (marked section-omitted), including parts of Section 4.1 and Appendices F/G; judgments rely on the surviving text.
- Figures 1-7 are assessed only through captions and in-text descriptions, not the rendered graphics.
- Repository contents at github.com/criticaldata/MODALENS were not verified; reproducibility claims are taken from the paper's statements.
- Some referenced appendix tables (e.g., Tables 13, 15, 32) are not fully present in the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.