MedQA-MM: Shortcuts Behind Medical Visual Reasoning
Benlu Wang, Yifan Zhang, Jiaqing Yu, Chin Siang Ong, Juncheng Huang, Zhuohao Li, Zhenyu Zhang, Arman Cohan, et al.
cs.CV · cs.CL
Quick verdict
Audits six medical MCQ benchmarks for shortcut cues inflating VLM scores; strong matched-repair evidence, but repairs are GPT-authored; expert review sampled.
Summary
The paper audits six widely used medical multimodal MCQ benchmarks (AMBOSS image questions, JAMA Clinical Challenge, MMMU-HM, MedThinkVQA, MedXpertQA-MM, NEJM Image Challenge; 7,706 items) for construct-irrelevant shortcut cues that let models score without image-grounded reasoning. On a 13-configuration open-model panel (MedGemma and Qwen families), full-input accuracy averages 62.63%, but text-only reaches 53.96% and options-only 29.71%; an options-only diagnostic guesser beats random by 15.99 pp. Matched repairs — minimal GPT-5.4 edits that preserve stem, image, and answer key — show behavioral cue dependence: removing length-gap cues cuts accuracy 6.58 pp (306 pairs), spatial/prepositional cues 4.77 pp (101), and absolute/conspicuous distractor cues 3.50 pp (101), supported by forward/reverse interventions, a text-identical negative control, and dose-response analysis. Image-hint removal (200 pairs) and clinical-context neutralization (302 pairs) show smaller, model-dependent effects. The authors assemble MedQA-MM, a 1,000-item shortcut-mitigated subset, where text-only accuracy falls to 5.21% and options-only to 12.33%; targeted error analysis finds item/data issues the largest class (29/70). Limitations: repairs and adjudication are largely LLM-authored (GPT-5.4, GPT-5.6-Sol) with expert validation sampled at ~246 cases; the subset's public release awaits license and image-rights clearance; and the headline panel excludes GPT configurations.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Full-input accuracy on six medical multimodal MCQ benchmarks mixes shortcut routes: in a 13-configuration open-model panel, text-only reaches 53.96% and options-only 29.71% versus 62.63% full-input.
- Matched cue-removal repairs causally reduce accuracy (-6.58, -4.77, -3.50 pp), demonstrating behavioral use of length-gap, spatial/prepositional, and absolute/conspicuous distractor cues.
- MedQA-MM, a 1,000-item shortcut-mitigated subset, weakens non-visual routes (text-only 5.21%, options-only 12.33%) but is shortcut-mitigated, not shortcut-free.
- Benchmark releases should include route-level audits, repair provenance, validation status, and residual-risk reporting to avoid 'reasoning inflation'.
Strengths
Causal controls rare in benchmark audits
Beyond plain ablations: forward destination analysis (edited distractor +9.47 pp [8.26, 10.75], Table 19), reverse key-shortening control (-6.79 pp in Options-only, Table 20), a 21-case text-identical negative control (-0.27 pp, CI spanning zero), and a 39-pair length-controlled spatial contrast (+2.70 pp, Table 22) triangulate that cue removal, not generic difficulty, drives the drops.
Calibrated claim discipline
Construct relevance is treated item-level (Section 3.1); MedQA-MM is explicitly 'shortcut-mitigated, not shortcut-free'; Image+Options exceeding full input by 1.88 pp (28.47% vs 26.59%) is reported as a caution against over-reading any single setting, and the error taxonomy finds item/data issues the largest class (29/70, Table 2).
Scale with disclosed bookkeeping
Actionable artifact plus reusable taxonomy
Weaknesses
LLM-in-the-loop repair and judging
Flagship subset not yet independently verifiable
Panel composition and parser conventions complicate synthesis
Control subsets are thin outside the length family
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether medical multimodal MCQ benchmark scores measure image-grounded clinical reasoning or a mixture of shortcut routes. It contributes (1) a route-level audit of six datasets (7,706 canonical items), (2) matched-repair causal evidence for three option-form cue families, and (3) MedQA-MM, a 1,000-item shortcut-mitigated subset. The construct is explicitly item-level: arrows, devices, and demographics are not invalid per se (Section 3.1). 'Reasoning inflation' is defined as score-level overinterpretation, not a claim about hidden cognition, and the paper repeatedly disclaims 'shortcut-free' status for its own artifact.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'reasoning inflation' in this paper?
The paper's term for score-level overinterpretation: benchmark accuracy is read as evidence of image-grounded medical reasoning even when some credited answers can be reached via construct-irrelevant routes such as option length, absolute wording, non-visual text, visible image labels, or annotations.
What is MedQA-MM and how hard is it?
A 1,000-item shortcut-mitigated subset built from MedThinkVQA, MedXpertQA-MM, and MMMU-HM (retaining 22.6% of 4,432 candidates). Under the tested probes, text-only accuracy falls to 5.21% and options-only to 12.33%, versus full-input 26.59% and Image+Options 28.47%. It is described as shortcut-mitigated, not shortcut-free, and its public release still awaits license and image-rights checks.
Do the results mean VLMs never use medical images?
No. Adding the image helps: the open-model panel gains about +14.13 pp going from options-only to Image+Options, and full input reaches 62.63% versus 29.71% options-only. The claim is narrower: scores on unaudited benchmarks cannot isolate image-grounded reasoning without route-level evidence.
How large are the measured shortcut effects?
Removing length-gap cues cuts accuracy 6.58 pp across 306 repaired pairs; spatial/prepositional cue removal cuts 4.77 pp (101 pairs); absolute/conspicuous distractor repair cuts 3.50 pp (101 pairs), each averaged over a 17-configuration panel with 95% CIs excluding zero.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Sampled excerpt: extraction skipped middle sections (marked '[...sampling gap...]'), including part of the Section 5 narrative and pieces of Appendices A, C.3, D.1-D.2, and F; per-dataset modality inventories were not visible, so dataset-level judgments beyond Tables 16 and 23 are limited.
- The task metadata lists 8 authors while the paper header lists 10 (adding Hong Yu and Zonghai Yao, the corresponding author); the review follows the paper text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.