Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflicts

Summary

This paper studies how vision-language models (VLMs) resolve context-memory conflicts—when in-context information contradicts parametric knowledge—depending on whether the conflicting entity appears as text or an image. The authors curate a 37,970-pair counterfactual fact-retrieval benchmark spanning celebrities, buildings, and artwork, evaluate seven open-source VLMs (Gemma-3, Qwen2.5-VL, Ministral-3), and filter to instances where models can both recognize the entity and recall the parametric fact. Behaviorally, models prefer contextual answers for text entities but show substantially higher parametric-answer rates for image entities (e.g., Gemma-3-12B building-location gap of +80.80 points; Qwen2.5-VL-7B artwork-artist gap of +48.25), with Gemma-3-27B the notable exception showing no gap. Mechanistically, causal interventions support a suppression account: MLP ablation over entity tokens shifts visual-entity outputs toward the contextual answer far more than textual ones; masking entity-to-context attention induces a large parametric shift for text entities that is largely neutralized when downstream MLP activations are restored to clean values; and back-patching late-layer visual activations into layer 1 drives parametric rates toward the text baseline, consistent with late visual-to-text alignment arriving too late to permit suppression. Chain-of-thought prompting does not reliably close the gap, while replacing textual context with an image reduces parametric bias for visual entities when the same image is reused. Limitations include sole reliance on an 8B LLM-as-judge for scoring and filtering, small post-filter samples for some relations (e.g., architect, n=10), restriction to open-source models, and a controlled single-image mock-RAG setting.