Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Models
Rwiddhi Chakraborty (Oliver), Yinong (Oliver), Wang, Cheng Zhang, Fan Bai, Zhuoran You, Michael Kampffmeyer, Yong Jae Lee, et al.
cs.CV
Quick verdict
Frontier video models collapse (Gemini 74%→30%) when MCQ options become visual frames; compelling design, but small datasets—esp. 20-video CMD—limit statistics.
Summary
The paper argues that standard video-LLM benchmarks pose multiple-choice questions only in text, allowing models to exploit language cues, and asks what happens when the evaluation modality is visual. It introduces three vision-centric tasks whose answer options are frame sequences: temporal frame retrieval (1,108 questions from 321 Charades videos), video future prediction (300 Shapestacks clips), and causal memory distortion (20 Charades videos adapted from Strickland & Keil's 2011 event-completion study). Eleven models — Gemini 2.5 Pro, GLM-4.6V, Qwen variants, LLaVA-OV-72B, InternVL3, Kimi K2.6, mPLUG-Owl3 — are evaluated against 25%/50% chance baselines and a ten-person human study. Most models sit near chance on retrieval; Gemini leads future prediction (69.6% vs 80% human). On Charades, Gemini's 73.9% text accuracy collapses to 30.0% with visual options; VideoCon hard text negatives (71.5%) and prompt variants do not close the gap. Trace analysis attributes failures to option grounding (Gemini) versus query grounding (Qwen); signal-detection analysis shows Gemini's low specificity (33%) reflects a yes-bias, not poor sensitivity. Limitations: small datasets — the memory task has 20 videos and no significance tests — a 44% human retrieval score that questions distractor validity, and model-list inconsistencies.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Frontier video-language models show a large accuracy drop when MCQ options are visual frame sequences instead of text (e.g., Gemini 2.5 Pro: 73.9% text vs 30.0% visual on temporal frame retrieval).
- The text-visual gap is not an artifact of text distractor difficulty or prompt wording (Table 3 hard negatives; Table 5 prompt ablation).
- Failures stem from grounding rather than perception: Gemini mis-maps correct reasoning to the wrong option while Qwen ignores visual options entirely (Sec. 5.2).
- On causal memory distortion, Gemini hallucinates removed contact frames at high rates (specificity 33%), reflecting a liberal yes-bias rather than poor sensitivity — the first such evaluation in video LLMs.
- Visual-option evaluation complements text benchmarks and exposes weaknesses that text-based evaluation can inflate or hide.
Strengths
Directly visual evaluation interface with controlled text-vs-visual comparison
All three tasks pose questions with frame-sequence options drawn from the source videos (Sec. 3), enabling same-task modality contrasts: Gemini drops 73.87→30.00 on Charades retrieval (Tables 1/3), and the gap survives VideoCon hard text negatives (71.48, App. C) and prompt rewrites (App. H, Δ −0.43 to −0.46), ruling out two obvious confounds.
First causal memory distortion test for video LLMs, grounded in cognitive science
Sec. 3.3 adapts Strickland & Keil (2011) event completion with complete/incomplete × with/without-causal-implication × scrambled conditions; Table 4's signal-detection analysis (Gemini sensitivity 83% vs specificity 33%, d′=0.51, criterion c=−0.70) separates hallucination rate from response bias — a genuinely diagnostic design.
Confound awareness beyond typical benchmark papers
Interpretable failure analysis
Weaknesses
Human baseline near chance on the flagship retrieval task
CMD task is severely underpowered for its headline claims
Model roster and reporting inconsistencies
Opaque uncertainty methodology for closed models
No variance reporting for the MCQ tasks
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
When video-language models answer multiple-choice questions whose options are sequences of video frames rather than text descriptions, accuracy collapses: Gemini 2.5 Pro falls from about 74% (text options) to 30% (visual options) on a temporal retrieval task where chance is 25%. Most of the eleven evaluated models perform near chance across the three visual tasks, indicating current models reason far better through text than through raw vision.
What is causal memory distortion and why is it tested on AI models?
It adapts a 2011 cognitive-science experiment by Strickland and Keil: a video's moment of contact (e.g., a ball being kicked) is removed, and the viewer is asked whether a contact frame appeared. Humans often falsely remember seeing it when the remaining footage causally implies the contact. The benchmark tests whether models show the same hallucination, finding Gemini 2.5 Pro answers 'yes' excessively (specificity 33%) while Qwen is more conservative — reportedly the first such test in video language models.
How big are the benchmark datasets and is code available?
The benchmark totals 641 videos: 1,108 temporal-retrieval questions from 321 Charades videos (about 35,000 frames), 300 future-prediction questions from Shapestacks physics simulations (12,000 frames), and 20 manually curated Charades videos (60 yes/no probes) for the memory task. The authors state the JSON annotations, prompts, and a video-annotation tool are released or will be released with the paper; no repository URL appears in the reviewed text.
Did humans outperform the AI models on these tasks?
Yes on video future prediction (80% human vs 69.6% for the best model, Gemini 2.5 Pro) and causal memory distortion (80% vs 58-60%), but only 44% on temporal frame retrieval against a 25% chance level. The authors attribute the low human retrieval score to genuine task difficulty, though a skeptical reader may read it as evidence that some visual distractor options are not clearly wrong even to people.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1, 5, 6, 10 and 11 are bar/box charts whose exact values are only partially recoverable from the extracted text; per-model text-vs-visual deltas beyond those quoted in prose could not be verified.
- The provided metadata lists 7 authors while the extracted full text lists 11 (adding F. De la Torre and R. Jenssen); this review follows the full text.
- Minor extraction artifacts exist (e.g., a JSON span rendered as '15 15.2'); all headline numbers were cross-checked against Tables 1-5 where possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.