Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Models
Shuzhi Gong, Fengze Sun, Yuansan Liu
cs.CV · cs.AI
Quick verdict
Benchmark scores can't locate hallucination origins in video agents; this rigorous 60k-run causal intervention study supports that, though on one dataset.
Summary
The paper asks whether existing video benchmarks can diagnose where hallucinations originate in multi-stage video agents that decompose understanding into grounding, watching, and reasoning. Observationally, the authors organize five benchmarks around these stages and evaluate three open-weight agents (Video-R1, VideoMind-7B, EVA), finding inconsistent diagnostic signals — including anomalous near-zero ELV-Halluc scores — that correlational cross-benchmark comparison cannot resolve. They then introduce a causal stage-intervention protocol: on 2,522 CG-BENCH questions over 227 long videos, they overwrite one stage while holding the downstream task fixed, across 60,008 runs. Replacing a random window with the oracle span improves accuracy by 16.6–17.9 points on all three architectures — roughly four times the effect of corrupting half the frames — yet deployed grounders capture only about one quarter of this headroom. Accuracy is insensitive to dilating the oracle window up to 8×, explaining why mIoU poorly predicts downstream utility, and wrong evidence proves more harmful than missing evidence. Auditing benchmarks against these interventions, no score detectably predicts causal cascade sensitivity, and DR.V diagnosis shows 42–65% of correct answers are unsupported by the video while the metric saturates under distribution shift. Limitations: causal results rest on a single dataset, one checkpoint per architecture conflates architecture with model identity, and the system-level audit (n=9) is underpowered.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing stage-aligned benchmark scores provide inconsistent signals and cannot attribute downstream hallucination to a specific pipeline stage in multi-stage video agents.
- Grounding causally dominates downstream error: substituting the oracle evidence span for a length-matched random window improves accuracy by 16.6–17.9 pp across three architectures, roughly 4× the effect of corrupting half the watched frames.
- Downstream accuracy is insensitive to oracle-window dilation up to 8×, so overlap metrics such as mIoU poorly predict downstream reliability.
- Incorrect evidence is substantially more harmful than missing evidence.
- No existing benchmark score detectably predicts causally measured cascade sensitivity, and 42–65% of correct answers are unsupported by the video.
Strengths
Causal design replaces cross-benchmark correlation
Section 3 overwrites exactly one pipeline stage on a fixed 2,522-item CG-BENCH set shared by all three agents, with pre-specified directional sanity checks (oracle > random; off-target ≯ random) that pass (Table 3) — a clean conversion of an attribution question into a measurable one.
Statistical rigor unusual for this literature
95% paired bootstrap CIs clustered by video (227 clusters), exact McNemar with Holm correction per contrast family, abstention tracking (≤1.7%, Appendix D), and BCa intervals for the audit correlations (Appendix C).
Actionable, non-obvious findings
Cross-architecture replication of the headline effect
Protocol transparency and honest limitations
Weaknesses
Single-dataset causal core
Architecture vs model identity confound
Universal negative claim rests on an underpowered leg
Intervention ecological validity
Presentation defects
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a diagnosis problem, not a modeling problem: can existing benchmark scores say *which stage* of a multi-stage video agent (Grounding → Watching → Reasoning) caused a downstream hallucination? Three claims: (1) fragmented stage-aligned benchmarks give inconsistent signals (Section 2, Table 2); (2) a causal stage-intervention protocol (Section 3) measures stage effects directly; (3) against this causal reference, existing scores fail as diagnostics (Section 4). Headline findings: grounding dominates (+16.6–17.9 pp oracle−random across three architectures, Table 3); on-target beats precise (dilation plateau to 8×, Table 5); wrong evidence hurts more than missing evidence (Table 4); 42–65% of correct answers are evidentially unsupported (Figure 4). The TAE-workshop framing matches the 'auditing evaluation itself' contribution.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of the paper 'Beneath the Scores'?
That standard video benchmark scores cannot tell you which stage of a multi-stage video agent causes hallucinated answers. Using a causal intervention protocol with 60,008 runs on three agent architectures, the authors show grounding errors dominate: fixing the grounded time window is worth roughly 17 accuracy points, about four times the effect of corrupting half the observed frames.
Why do the authors argue mIoU is a poor grounding metric?
Because downstream accuracy barely drops when the correct evidence window is dilated up to 8× (about 5 minutes around a 10-second clue), even though such dilation sharply reduces temporal overlap scores like mIoU. Being on the right region of the video matters more than having precise boundaries.
What does '42–65% of correct answers are unsupported' mean?
Using the DR.V evidence-level diagnosis, the authors found that a large share of answers counted as correct by accuracy metrics are not actually justified by the video frames the agent saw. Final-answer accuracy therefore overstates true reliability.
Is code or data available for this paper?
Not at submission time. The checklist states code and instructions will be released upon acceptance, though the paper describes seeded, deterministic span manifests and per-run logs designed for exact re-runs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–6 are available only as extracted captions and axis labels; visual content (e.g., Figure 3 regression slopes) could not be independently inspected.
- Appendix sections G–K appear partially truncated or misordered in the extraction (e.g., Table 11 surfaces before Tables 8–10).
- Appendix A.2 and A.3 are verbatim duplicates in the source text — likely a typesetting error in the paper itself.
- Several references are duplicated (Chen 2025a/b, Rawal 2025a/b, Wang 2024b/c, Xiao 2024a/b), likely double-blind artifacts.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.