Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?
Zhaoyang Wei, Zipeng Wang, Yushe Cao, Chenhui Qiang, Shuaibing Cheng, Xuesong Yang, Sen Nie, Bowen Jiang, et al.
cs.CV
Quick verdict
Rigorously built audio-visual evidence benchmark + token compression, but abstract numbers conflict with its own tables and gains lack significance tests.
Summary
The paper introduces Video-HolmesV2, a benchmark of 784 post-2024 cinematic videos (avg ≈43 min) with 4,000 multiple-choice questions that can only be answered by coupling visual and auditory evidence, plus a required spatio-temporal evidence chain. Questions are generated by long-context MLLMs (stratified by video length, seeded from user commentary for films), filtered by a 'unimodal trap' (gpt-4o-mini must fail with either modality alone), cross-verified by a model consortium requiring timestamp IoU > 0.75, and hard cases adjudicated by double-blind human experts. Evaluation combines binary answer accuracy with a Gaussian soft temporal-alignment score and an F0.5 LLM-judged semantic score, rewarded only when the answer is correct. Results: Gemini-2.5-Pro leads (73.2 overall; evidence score 35.2/100); the best open model, Qwen3-VL-235B, scores 55.3 with a near-zero evidence score (8.5), which the authors read as memorized guessing. The companion Audio-Text Guided Token Compression (a 0.5B rank-distilled text selector fused with audio-saliency anchors under a fixed 8,192-token budget) lifts a 7B omni-model to 44.5 and slightly beats full-token inference on VideoMME/WorldScene (70.8 vs 70.5). Main limitations: abstract/table inconsistencies, no significance testing, uncertain evidence-metric interpretability, baseline modality inequities, and no code/data release.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Video-HolmesV2 is the first benchmark requiring MLLMs to justify multiple-choice answers on hour-long videos with precise spatio-temporal audio-visual evidence.
- High MCQ accuracy coexists with near-zero evidence quality (Qwen3-VL-235B: 55.1 accuracy vs 8.5 evidence), showing conventional leaderboards reward ungrounded guessing.
- The Audio-Text Guided Token Compression framework preserves reasoning under a fixed 8,192-token budget and slightly exceeds full-token inference on external benchmarks (70.8 vs 70.5 average).
- Multi-model cross-verification with unimodal-trap filtering and expert adjudication yields a benchmark where even Gemini-2.5-Pro reaches only 73.2 overall with 35.2/100 evidence quality.
Strengths
Fills a genuine gap in evaluation design
Table 1 shows no existing A+V benchmark combines hour-long durations with spatio-temporal evidence and evidence chains; Video-HolmesV2's 784 videos / 4,000 QA at 600–7500 s with mandatory timestamped audio-visual grounding is a real first along that axis.
Multi-stage adversarial QC pipeline
The 'Unimodal Trap' (gpt-4o-mini must fail on visual-only and audio-only inputs), ensemble verification demanding answer consensus plus timestamp IoU > 0.75 (Sec. 3.2, Fig. 3), and double-blind expert adjudication with κ ≥ 0.8 arbitration (App. A.3) is far more rigorous than typical LLM-generated benchmarks.
Answer-conditioned evidence metric
Judge validation above community norm
Compression generalizes out of domain
Weaknesses
Abstract contradicts the results tables
Evidence-score collapse is under-diagnosed
No uncertainty quantification
Baseline comparability is compromised
No release path and residual contamination risk
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Video-HolmesV2 (arXiv 2609.17248) is a dual-contribution paper: (1) a long-form audio-visual QA benchmark — 784 post-2024 cinematic videos, 600–7500 s, avg ≈43 min, 4,000 MCQ items (Table 1) — whose defining constraint is *Evidence-Based Evaluation*: a correct answer must be accompanied by precise spatio-temporal audio-visual evidence; and (2) an Audio-Text Guided Token Compression method (TGC + AGC + DSFDC) that pushes a 7B omni-model through hour-long video under a fixed 8,192-token budget. Central claims: current evaluation marginalizes audio; models guess answers without grounding (Sec. 6.3); the proposed compression beats full-token and ratio-based baselines (Tables 4–5). The abstract's headline numbers, however, do not match the results tables (see Risks).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Video-HolmesV2 benchmark?
A research benchmark of 784 long videos (10–150 minutes, released after 2024) with 4,000 multiple-choice questions that require combining what is seen and what is heard. Unlike earlier benchmarks, models must also output the exact timestamps of the visual and audio evidence supporting their answer, and evidence only counts if the final answer is correct.
How well do current AI models perform on Video-HolmesV2?
Google's Gemini-2.5-Pro leads with an overall score of 73.2, but its evidence score is only 35.2/100. The best open-source model, Qwen3-VL-235B, answers 55.1% of questions correctly yet scores just 8.5/100 on evidence, which the authors interpret as answering from memorized plot knowledge rather than the video. GPT-4o scores 46.9, partly due to a 50-frame input cap.
What does the paper's token compression method do?
It lets a 7B omni-model watch hour-long videos within a fixed 8,192-visual-token budget. A tiny 0.5B network distilled from a 72B teacher ranks visual tokens by query relevance; a training-free audio scorer highlights regions aligned with salient sounds; redundant core tokens are merged and the saved budget is spent restoring background context. It slightly beats uncompressed full-token inference on VideoMME (70.8 vs 70.5).
Is the Video-HolmesV2 data or code publicly available?
The paper contains no code or data release statement. The videos are copyrighted films, series, and festival shorts, so public distribution is legally complicated; at present the benchmark, prompts, and judge configuration are not confirmed as available.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (Fig. 2 taxonomy distribution, Figs. 5–9 case studies) are not analyzable from the extracted text; assessments of qualitative examples rely on captions and prose only.
- Pipeline yield statistics (per-phase discard rates, Hard Subset size, genre/duration split of the 784 videos) are referenced to Appendix A but not fully present in the extracted text.
- The Fig. 1 region contains OCR artifacts and typos (e.g., 'EVIDENEC', inconsistent timestamps), so the flagship worked example could only be assessed approximately.
- The metadata author list (9 names) is shorter than the full byline in the extracted text (14 names); review assumes the in-text byline is authoritative.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.