Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict
Jungyeon Lee, Yejin Yoon, Taeuk Kim
cs.CL
Quick verdict
Claims same evidence as image vs text flips MLLM conflict resolution; thorough 13-model study, but no significance tests and mitigations mostly fail.
Summary
This empirical study examines whether multimodal LLMs handle semantically identical evidence consistently when it conflicts with their parametric knowledge. The authors construct single-evidence conflicts (the same counterfactual passage given as text or as a rendered image) and multi-evidence conflicts (text and image supporting different answers) on CONFLICTQA and NQ-SWAP, evaluating 13 MLLMs with a strict protocol: parametric knowledge is estimated via five unanimous closed-book samples per model, and the external-source following rate (EFR) requires three unanimous generations. Contrary to the widely reported text bias, models follow image-rendered evidence more readily in the single-evidence setting — most strongly for proprietary models (Claude Sonnet 4.5: +42.86 normalized gap on CONFLICTQA) — while many open-source models show near-zero gaps. With both modalities present, preference is unstable, flipping with input order (8/13 models change sign on CONFLICTQA), model family, and dataset. Downstream, image-rendered gold passages mitigate lost-in-the-middle in RAG (+6.59 ROUGE-L for Gemini 2.5 Pro in middle position, 170 MS MARCO instances), and rendering harmful instructions as images raises attack success rates by 6.99 points on average across 156 settings. Prompting, steering, and DPO shift rather than balance the bias; conflict-aware SFT on Qwen2.5-VL-7B reduces gaps moderately. Main limitations: no significance testing, per-model evaluation subsets, and mitigation tested on a single model.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under single-evidence knowledge conflict, MLLMs follow evidence rendered as an image more readily than the same evidence given as text, reversing the commonly reported text bias (strongest in proprietary models, e.g., Claude Sonnet 4.5 +42.86 on CONFLICTQA).
- When contradicting text and image are presented together, modality preference is unstable and flips with input order (8/13 models change the sign of G on CONFLICTQA), model family, and dataset.
- Modality instability has practical consequences: image-rendered gold passages mitigate lost-in-the-middle in RAG (+6.59 ROUGE-L for Gemini 2.5 Pro in the middle position), and rendering harmful instructions as images raises ASR by 6.99 points on average across 156 settings.
- Prompting, representation steering, and DPO shift rather than balance modality preference; conflict-aware SFT is the only intervention that moves all gap metrics toward zero while selectively reducing misleading-image following.
Strengths
Strict parametric-knowledge control
§3.1 pins each model's internal answer via five unanimous closed-book samples (stricter than Wang et al. 2023's majority vote), with 'none of the above' fallback, and reports per-model retained counts (Tables 6-7), directly addressing the confound flagged in prior multimodal KC work (footnote 2).
Conservative metric design
EFR (§3.2) credits evidence-following only when three sampled generations unanimously match the external answer, and the normalized gap G ∈ [-100, 100] enables cross-model comparison despite differing overall following rates.
Systematic confound elimination
Causal input-side probe
Honest mitigation reporting
Weaknesses
No inferential statistics
Per-model instance subsets
Abstract overgeneralizes claim (1)
Mitigation tested on one model
RAG probe is thin and roster-inconsistent
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper formalizes modality robustness as an evidence-authority question under knowledge conflict (KC): does an MLLM's answer change when the same contradicting evidence is delivered as text versus as a rendered image of that text (§3)? Three central claims: (1) under single-evidence conflict, models follow image-form evidence more readily than text — reversing the "blind faith in text" consensus (Deng et al., 2025); (2) under multi-evidence conflict, modality preference is unstable, flipping with input order, model family, and dataset; (3) prompting, steering, and DPO largely fail while conflict-aware SFT moderately helps. Downstream stakes are demonstrated in multimodal RAG (§5.1) and safety (§5.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
Multimodal LLMs treat semantically identical evidence differently depending on its surface form. Under knowledge conflict, models follow evidence rendered as an image more readily than the same text, and when conflicting text and image appear together, which modality 'wins' flips with input order, model family, and dataset.
Does this contradict the reported 'text bias' in vision-language models?
Yes, in this setting. With parametric knowledge strictly controlled via unanimous closed-book sampling, the single-evidence image preference is clearest in proprietary models (Claude Sonnet 4.5 shows a +42.86 normalized gap on CONFLICTQA), though many open-source models show near-zero gaps, so the reversal is not universal.
Can the instability be fixed?
Only partially. Explicit prompting, activation steering, and DPO mostly shift the preference toward one modality rather than balancing it. Conflict-aware supervised fine-tuning on Qwen2.5-VL-7B reduced gap magnitudes in all tested settings (e.g., 73.82→52.04 in one order) and lowered following of misleading images while preserving use of reliable ones.
Is the effect just because models misread rendered text in images?
No. Transcription of the rendered images achieves near-perfect ROUGE-L (≥98.37), and changing font, color, or background barely shifts results; only highlighting the answer span moves behavior substantially, pointing to cross-modal integration rather than perception as the source.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with marked gaps; portions of §3.1, §6.1, Appendix C.1, and Appendix G were not visible, so some details (e.g., full SFT pipeline and ASR judging) may exist in the full paper.
- No significance testing was visible in the provided text; if tests exist in unsampled sections, the statistical critique should be softened accordingly.
- Figure 1's inline text was partially garbled by PDF extraction, though its caption and the surrounding prose fully convey the setup.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.