AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents
Seoyeon An, Hyeonseo Jang, Minsu Kim, Chanho Lee, Younghan Park, Younghan Park, Kangwook Lee
cs.CV · cs.AI
Quick verdict
Introduces a 100-question multi-hop video QA benchmark with LLM-judged trajectory scoring; judge validation is solid, but scale and self-baseline limit reach.
Summary
The paper introduces AgentVidBench, a benchmark of 100 multiple-choice questions over 71 openly licensed videos (8 seconds to 27 minutes, 12 genres) whose questions are authored to require multi-hop spatial, temporal, and causal reasoning rather than single-frame lookup. Each question carries 26 answer options (chance 3.85%), a three-annotator blind consensus answer (Fleiss' κ 0.826), and tool-agnostic evidence milestones abstracted from human reasoning notes, enabling an LLM judge to score agent trajectories along five axes (task understanding, evidence coverage, grounding, completeness, faithfulness). Twelve MLLMs (GPT-5, Claude Opus 4.7, Gemini 2.5, Qwen3-VL, Qwen3.5, Gemma 4) are evaluated single-turn and inside four agentic frameworks (STAR, DVD, AVP, and the authors' Video-ReAct). Single-turn accuracy tops out at 0.51 (Gemini 2.5 Pro); Video-ReAct reaches 0.54–0.58 with the strongest backbones; some configurations, notably STAR, fall below single-turn. Judge validation is unusually thorough: correct-vs-incorrect separation AUC 0.93–0.98, adversarial and milestone-ablation tests, human-judge correlation r=0.77, plus paired bootstrap CIs, pooled McNemar tests (p<0.001), and ranking stability (Kendall τ-b 0.83). Main limitations: small scale (100 questions), difficulty tiers calibrated post hoc from model performance, reliance on closed models for judging and milestone extraction, and a perception tool fixed to Gemini 2.5 Pro, which complicates attributing weak backbones' gains to planning ability.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing video QA benchmarks require only single-step inference, so AgentVidBench authors 100 questions (26 options each, chance 3.85%) over 71 openly licensed videos that provably demand multi-hop evidence gathering across spatial, temporal, and causal dimensions.
- Tool-agnostic evidence milestones paired with an LLM judge produce a trajectory score that separates correct from incorrect runs (AUC 0.93–0.98), resists superficial gaming (adversarial trajectories score 0.096), and aligns with human ratings (r=0.77).
- Single-turn MLLMs are limited on the benchmark (best accuracy 0.51, Gemini 2.5 Pro), while integrating them into agentic workflows generally improves both accuracy (best 0.58, Video-ReAct) and trajectory scores, with some frameworks (STAR) underperforming single-turn.
- Adaptive video-inspection controls (FPS, resolution, time interval) each contribute to agentic performance (Table 2), and agentic methods beat single-turn in token efficiency (AVP 5.28% vs 3.36% accuracy per MTok, Table 3).
- On Video-MME Long, single-turn and agentic Gemini 2.5 Pro both reach ~90%, indicating existing single-hop benchmarks saturate and cannot expose agentic gains (Appendix I).
Strengths
Exemplary judge validation suite
Appendix D.4 validates the LLM trajectory judge four ways: correct-vs-incorrect run separation at AUC 0.93–0.98 across three judge models; synthetic confident-answer trajectories score 0.096 vs 0.758 for genuine ones (AUC 1.000); removing one milestone's supporting evidence drops scores by 0.41 with 73% of coverage labels changing; human-judge Pearson r=0.77 on 100 blind-rated trajectories.
Statistical reliability analysis rare for benchmark papers
Appendix G adds paired bootstrap 95% CIs over 10,000 resamples (Table 20, 8/12 intervals exclude zero), pooled McNemar tests across 16 open-weight settings (all p<0.001, Table 21), and ranking stability with Kendall τ-b 0.83 plus bootstrap rank intervals (Table 22).
Careful, honest annotation and controls
Actionable diagnostic depth
Saturating-baseline evidence for motivation
Weaknesses
Small scale limits granularity and some key comparisons
Planner-vs-perceiver attribution confound in Video-ReAct
Closed-model dependence inside the evaluation loop
Post-hoc difficulty calibration is partially circular
Decorative theory and self-favored baseline
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AgentVidBench (arXiv 2609.21386, KRAFTON) targets a real gap: video benchmarks score final answers, not the evidence-gathering process of MLLM agents. The authors build 100 multiple-choice questions over 71 openly licensed videos (8 s to 27 min, median ≈5.5 min; 12 genres in 4 super-categories), explicitly prohibiting questions solvable from a single frame or one timestamp lookup. Every question has 26 options (chance 3.85%), a blind three-annotator consensus answer (Fleiss' κ 0.826; 22 questions flagged, 7 keys corrected — C.2), and 3–7 tool-agnostic evidence milestones abstracted from human reasoning notes (456 total; temporal localization 24.8% most frequent). Central claims: (1) single-turn MLLMs are limited (best accuracy 0.51, Gemini 2.5 Pro, Table 1); (2) agentic workflows generally improve accuracy and trajectory scores (their Video-ReAct reaches 0.54–0.58 accuracy, up to 0.79 trajectory); (3) the milestone-based trajectory judge is a validated, path-independent process metric. The hedge *generally* is honest: Gemini 2.5 Flash loses 0.01 accuracy under Video-ReAct (0.46→0.45), and STAR underperforms single-turn for most backbones (e.g., 0.09 vs 0.22 for Qwen3.5-27B).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AgentVidBench?
A benchmark of 100 multiple-choice video questions over 71 openly licensed videos (8 seconds to 27 minutes), each with 26 answer options (random guessing scores 3.85%). Questions require multi-hop spatial, temporal, and causal reasoning that cannot be solved from a single frame, and each comes with human-written evidence milestones so that AI agents' reasoning trajectories — not just final answers — can be scored.
How is AgentVidBench different from Video-MME or other video benchmarks?
Video-MME-style benchmarks measure final-answer accuracy of models watching the whole video and are near saturation (~90% on the Long subset for Gemini 2.5 Pro). AgentVidBench instead requires iterative tool use (zooming into time ranges, adjusting FPS/resolution, reading transcripts) and adds a milestone-based trajectory score that grades whether the agent actually gathered the evidence needed for its answer.
Which models perform best on AgentVidBench?
Single-turn, Gemini 2.5 Pro leads at 0.51 accuracy (GPT-5 0.29, Claude Opus 4.7 0.30). Inside the authors' Video-ReAct agentic loop, the best configurations reach 0.54–0.58 accuracy and trajectory scores up to 0.79 (Claude Opus 4.7 backbone). Some agentic frameworks, notably STAR, perform worse than plain single-turn inference.
Is the AgentVidBench code and data publicly available?
The paper states code and datasets are released at github.com/krafton-ai/agentvidbench and huggingface.co/datasets/agentvidbench/agentvidbench, and the full judge and milestone-extraction prompts plus model settings are printed in the paper's appendices. Note that reproducing all results requires paid access to proprietary models (GPT-5, Claude Opus 4.7, Gemini 2.5).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Radar-plot figures (Figs. 4, 6–8) and filmstrip examples are not numerically analyzable from the extracted text; per-axis trajectory values rely on prose descriptions.
- Code/data repository existence and contents were not verified (no network access in this review); availability is as stated by the authors.
- Traj score reproduction will depend on specific proprietary API versions (GPT-5 extractor, Opus 4.7 judge), which may drift after publication.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.