ECG Mirage: Revealing and Mitigating the Underutilisation of ECGs in Vision-Language Models for Clinical Prediction
Jinning Liang, Mingcheng Zhu, Tingting Zhu
cs.AI
Quick verdict
Claims VLMs ignore patient ECGs; visual-prompt tuning with conditional DPO restores ECG use, but trails LoRA on accuracy and statistics are bootstrap-only.
Summary
The paper names and probes 'ECG Mirage': emergency-department VLMs can appear multimodal while gaining nothing from the correct patient's ECG. Two failure modes are formalised — ECG neglect (matched ECGs no better than mismatched or none) and ECG confusion (better than none, not better than mismatched) — and tested by holding EHR text, question, and target fixed while swapping in matched ECGs, outcome-discordant mismatched ECGs, or no image. On MDS-ED (built from MIMIC-IV-ED with linked ECGs), four zero-shot VLMs (Qwen3.5-4B, Qwen3.8-27B, Gemma4-4B, MedGemma1.5-4B) show no consistent matched-ECG advantage for ICU admission or six-label deterioration prediction, with balanced accuracies near chance (52-58%). As mitigation, the authors train four restricted visual prompts on a frozen Qwen3.5-4B — text tokens are masked from attending to the prompts — with supervised fine-tuning followed by conditional DPO that favours correct answers under matched over mismatched ECGs. Matched balanced accuracy rises to 70.6% (ICU admission) and 67.5% (deterioration), with matched-versus-mismatched gaps of 16.5 and 5.5 percentage points; ablations show DPO drives the separation. Limitations: LoRA still attains higher matched accuracy (74.7%/69.2%) without modality dependence, the zero-shot models sit near the performance floor, only one backbone is adapted, and uncertainty is limited to bootstrap standard deviations without significance tests.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across four zero-shot VLMs on MDS-ED, matched ECGs provide no consistent predictive advantage over outcome-discordant mismatched ECGs or no-image controls for ICU admission or six-label deterioration prediction (ECG Mirage).
- Restricted visual prompt tuning (4 prompts, frozen backbone, text masked from prompts) with SFT then conditional DPO lifts matched balanced accuracy to 70.6% (ICU) and 67.5% (deterioration) on Qwen3.5-4B.
- The method widens matched-versus-mismatched gaps to approximately 16.5 and 5.5 percentage points, with conditional DPO identified by ablation as the component driving the separation.
- LoRA achieves higher absolute matched performance but similar scores across ECG conditions, demonstrating that task accuracy and patient-specific modality dependence are separable objectives.
Strengths
Clean controlled probe of modality dependence
Sec. 2.2/Fig. 1 hold EHR text, question, and target fixed while comparing matched ECG (I+), outcome-discordant mismatched ECG (I−), and no-image (∅) — separating 'an ECG helps' from 'the correct patient's ECG helps', a distinction most multimodal clinical evaluations miss.
Formal, falsifiable definitions
ECG neglect (R+ ≥ R− and R+ ≥ R0) and ECG confusion (R− ≤ R+ < R0) are stated as explicit expected-loss inequalities, making the phenomenon testable rather than anecdotal.
Mechanism-level mitigation with ablations
Honest tradeoff reporting
Weaknesses
Floor-effect confound in the RQ1 diagnosis
Mitigation trades discriminative power for faithfulness
No inferential statistics
Single backbone for all adaptation results
Mismatch protocol underspecified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This 4-page conference-format paper asks whether vision-language models (VLMs) actually use the *correct patient's* ECG when predicting emergency-department outcomes from EHR text plus ECG images. It names the failure mode "ECG Mirage" and formalises two variants as expected-loss conditions over three input configurations (Sec. 2.2): ECG neglect (R+ ≥ R− and R+ ≥ R0) and ECG confusion (R− ≤ R+ < R0), where R+ is the loss with the matched ECG, R− with an outcome-discordant mismatched ECG, and R0 with no image — all with text, question, and target held fixed. Three claims are advanced: (1) four zero-shot VLMs on MDS-ED show no consistent matched-ECG advantage on ICU admission or six-label deterioration; (2) restricted visual prompt tuning (VPT) with supervised fine-tuning then conditional DPO restores patient-specific ECG benefit (matched-mismatched gaps of 16.5 and 5.5 pp); (3) every proposed component contributes (Table 2 ablations).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'ECG Mirage'?
A failure mode named in this paper: a multimodal model appears to use ECG images for clinical prediction, but its predictions would be essentially unchanged if given another patient's ECG — or no ECG at all. The paper splits it into 'ECG neglect' (the correct ECG adds no benefit over wrong or missing ECGs) and 'ECG confusion' (better than no image, but no better than a wrong patient's ECG).
How well does the proposed fix perform?
After training four restricted visual prompts on a frozen Qwen3.5-4B with supervised learning plus conditional preference optimisation (DPO), matched-ECG balanced accuracy reaches 70.6% for ICU admission and 67.5% for six-label deterioration prediction, with matched-versus-mismatched gaps of about 16.5 and 5.5 percentage points. However, LoRA still achieves higher absolute matched accuracy (74.7% and 69.2%).
What data and models does the paper use?
The MDS-ED benchmark built from MIMIC-IV-ED with linked MIMIC-IV and MIMIC-IV-ECG (PhysioNet, credentialed access), with roughly 109,000 training visits per task and patient-level train/test splits. Four zero-shot vision-language models are evaluated: Qwen3.5-4B, Qwen3.8-27B, Gemma4-4B, and MedGemma1.5-4B; all fine-tuning experiments use Qwen3.5-4B.
Is code or data available?
The paper links a public code repository (github.com/JasonZuu/ECG-Mirage). The underlying MIMIC data is not freely downloadable — it requires PhysioNet credentialing under a data use agreement, so exact replication requires approved access.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2 zero-shot values for Gemma4-4B, MedGemma1.5-4B and Qwen3.8-27B appear only as bars; only Qwen3.5-4B's ICU numbers are quoted in the text, so per-model figure values were not verifiable.
- The paper is a 4-page venue-format contribution; implementation details (prompt initialisation, image resolution/tokenisation, exact mask placement per layer) are thin, which limits reproducibility judgement.
- Only the extracted text was reviewed; no external verification of the GitHub repository contents was performed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.