From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making
Davide Testa, Hugh Mee Wong, Alessandro Lenci, Bernardo Magnini, Albert Gatt
cs.CL · cs.CV
Quick verdict
Attention-knockout study of video-to-text flow in two 7B VLMs: solid causal design, plausible findings, but no significance tests and only small models tested.
Summary
The paper investigates how vision-language models integrate visual evidence when choosing between two candidate video descriptions. Using caption-foil pairs from the Italian MAIA benchmark (1,600 pairs each for spatial, causal, and temporal reasoning; 100 videos), the authors apply Attention Knockout - blocking attention from textual tokens to visual tokens - in Qwen2.5-VL 7B and LLaVA-OneVision 7B, with layer-wise logit-lens readouts of caption-vs-foil log-probability differences. Three findings emerge: (1) visual information is integrated mainly while the model processes the candidate answer options, which act as primary grounding sites, whereas the question and assistant regions contribute little; (2) token-level masking shows nouns are the most affected category (semantic anchors), with verbs gaining importance for temporal items; (3) temporal reasoning behaves differently: full-sentence scoring drops accuracy below chance (0.43/0.44 vs 0.77/0.73 under A/B scoring), decisions resolve late in the network, and marker analysis reveals lexical biases (e.g., 'Before' favored in text-only conditions at 0.82/0.75 vs 'After' 0.42/0.36; 'While' at 0.13/0.17 even with full vision). Limitations: only two 7B models are tested, no statistical significance testing is reported, and the text-to-vision control runs on one model only under a non-standard prompt order.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Visual information is integrated primarily while the model processes the candidate answer options, which are the primary textual grounding sites, whereas the question and assistant regions contribute little.
- Nouns are the linguistic category most dependent on visual input (semantic anchors), while verbs become more prominent when temporal relations must be resolved.
- Temporal decisions are determined late in the network and are more fragile than spatial/causal ones, with the averaged late-layer foil preference driven by incorrect items (Appendix A.3).
- Below-chance Temporal performance under full-sentence scoring partly reflects lexical biases of temporal markers (e.g., While at 0.13/0.17 accuracy even with full video input, Table 4), compounding genuine difficulty maintaining sequential information across frames.
Strengths
Precise causal intervention design
Masking is applied directly to pre-softmax attention scores via (q, k_start, k_end) tuples, globally across layers, rather than via hook-based approximation (Section 3.3), following and extending Zhang et al. 2025b to video.
Multiple orthogonal controls
Text-only baseline, A/B-label vs full-sentence scoring comparison (Table 2), a reverse text-to-vision direction (Appendix A.1), English cross-lingual replication on 200-item samples (A.2), and correct/incorrect split analysis (A.3).
Honest handling of the Temporal anomaly
Clean grounding-site gradient
Two architecturally distinct model families
Weaknesses
No inferential statistics anywhere
Layer-wise narrative rests entirely on logit-lens readout
Mechanistic target diverges from canonical behavior on Temporal
Incomplete reverse-direction control
Small-model scope and tokenization/POS dependencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: From Vision to Language (arXiv 2609.05149)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Attention Knockout in this paper?
A causal intervention that sets pre-softmax attention scores to negative infinity for chosen (query, key-span) pairs during the forward pass, applied across all transformer layers. Here it blocks attention from text tokens to video tokens (or vice versa) so the authors can measure how losing access to the video changes the model's preference between a correct caption and a minimally different foil.
Which models and data were used?
Two 7B vision-language models, Qwen2.5-VL-7B-Instruct and LLaVA-OneVision (qwen2-7b), tested on 4,800 Italian caption-foil pairs from the MAIA benchmark - 1,600 each for spatial, causal, and temporal relations, linked to 100 videos, with each video sampled into 16 frames.
Why do the models fail on temporal questions?
Two compounding reasons: models struggle to maintain event order across frames (temporal decisions resolve only in the final layers and flip easily), and the sentence-scoring setup is biased by temporal markers - for example, items containing 'While' score at only 0.13-0.17 accuracy even with full video input, and text-only models strongly prefer captions with 'Before' (0.82/0.75) over 'After' (0.42/0.36).
Is code and data available for this paper?
The paper states that code and data are available on GitHub, but no concrete repository URL appears in the provided text; the models (Qwen2.5-VL-7B, LLaVA-OneVision-7B) and the MAIA benchmark are publicly available on Hugging Face per Appendix C.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The heatmap figures (Figures 2-5, 7, 8, 12-14) are present only as garbled axis/tick text in the extraction, so layer-wise effect magnitudes could be assessed only through the in-text descriptions and Tables 2-4, not by reading exact cell values.
- The sentence 'Code and data available on Github' carries no repository URL anywhere in the extracted text; availability is taken at face value as stated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.