VidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Duration
Changbeen Kim, Junwon Chang, Kipyo Kim, Risa Shinoda, Kuniaki Saito, Donghyun Kim, Donghyun Kim
cs.CV · cs.AI
Quick verdict
500-video event-verification benchmark exposing Video-LLM hallucination and self-bias; rigorous design, but κ=0.50 annotator agreement warrants caution.
Summary
The authors introduce VidOmni-Bench, a sentence-level event verification benchmark for Video-LLMs. For each of 500 videos spanning five spatio-temporal complexity types and durations from 4 seconds to 90 minutes, dense captions are generated by five Video-LLMs (InternVL3.5 38B, Qwen3-VL 32B, video-SALMONN 2+ 72B, Keye-VL 1.5 8B, Gemini 3 Flash), and human annotators label each of ~64K sentences as Correct, Incorrect, or Unknown; incorrect sentences serve as hard negatives. Models must judge each caption sentence against the video, evaluated by per-video Precision/Recall/F1. Results: the best F1 is Gemini 3 Flash with audio at 41.7%; the best open model, Qwen3-VL 32B, reaches 21.7%, while several models fall below 10%. Reported findings include high hallucination rates in captioning (video incorrect rates 85.6–94.0%), a precision-recall gap, scaling gains concentrated in static-frame errors, self-preference bias (Gemini's F1 roughly halves on its own captions), and audio helping Gemini but hurting video-SALMONN 2+. Strengths include controlled complexity/duration axes and leakage checks; limitations include moderate inter-annotator agreement (Fleiss' κ=0.50), author adjudication of disputes, model-generated (hence generator-specific) negatives, Certificate Coverage estimated on only 30 videos per benchmark, and no public redistribution of the video files.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- VidOmni-Bench provides 500 videos (4 s to 90 min) with ~64K human-verified sentence-level labels for verifying dense video captions across controlled complexity and duration axes.
- Current Video-LLMs hallucinate frequently in dense captioning (video incorrect rates of 85.6-94.0%, Table 1) and are unreliable verifiers of plausible-but-wrong sentences (best F1 41.7%, most models far lower, Table 3).
- Video-LLMs show self-preference bias, performing markedly worse on their own captions (Gemini 3 Flash: F1 above 40 cross-source vs 21.9 self-verification, Fig. 12).
- Scaling model size mainly improves verification of static-frame errors while temporal errors remain bottlenecks, and weaknesses are model-specific across complexity and duration (Figs. 10-11).
Strengths
Controlled two-axis design disentangles complexity from duration
500 videos balanced across five complexity types and three duration brackets (100 per type, Fig. 4), from public datasets plus CC-licensed YouTube (Tables 4-5) — a cleaner diagnostic grid than the MCQA/GT-matching benchmarks critiqued in Table 2.
Large human-verified hard-negative pool with a structured error taxonomy
~64K sentence labels over 2,500 video-caption pairs; pilot-screened annotators, ≥2 annotators per pair, batch audits every 100 pairs, and a 10-category word-level taxonomy splitting static-frame vs temporal errors (Table 7, App. B.3-B.4).
Rich ablation and leakage controls beyond the main table
Evidence-coverage positioning quantified
Weaknesses
Ground-truth reliability only partially verifiable
Generator-specific negatives create partial circularity
Thin statistical support for headline rankings
Confounded evaluation configurations
Minor design inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
VidOmni-Bench (arXiv:2609.21521) is a benchmark paper targeting fine-grained video understanding in Video-LLMs. The core task is sentence-wise event verification: given a video and an n-sentence dense caption, a model labels each sentence as supported (0) or hallucinated (1) (§3.1, Eq. 1). The benchmark spans 500 videos over five spatio-temporal complexity types (Static World through Multi-Scene Dynamics, Table 4) crossed with three duration brackets (≤3 min, 4–30 min, ≥30 min; 100 videos per type, Fig. 4). Captions are generated by five Video-LLMs (InternVL3.5 38B, Qwen3-VL 32B, video-SALMONN 2+ 72B, Keye-VL 1.5 8B, Gemini 3 Flash) under five prompt templates, yielding 2,500 video-caption pairs and ~64K human-verified sentence labels (Table 1). The authors claim (i) pervasive captioning hallucination (video incorrect rates 85.6–94.0%), (ii) weak verification ability, and (iii) model-specific weaknesses across complexity and duration. The presented data broadly support these claims.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is VidOmni-Bench?
A benchmark of 500 videos (4 seconds to 90 minutes) where AI video models must check, sentence by sentence, whether a dense video description is actually supported by the video. Sentences shown by human annotators to contain wrong events serve as hard negatives that models should catch.
How well do current video AI models perform on it?
Poorly. The best score is Gemini 3 Flash with audio at 41.7% F1; the best open-source model, Qwen3-VL 32B, reaches 21.7%, and several models score below 10%, mostly because they miss hallucinated sentences rather than flagging correct ones wrongly.
Is the dataset publicly available?
Partially. The authors will release video metadata, the generated captions, and the human annotations, but not the video files themselves (for copyright reasons), so users must re-download the 500 videos from the original public sources.
What is the self-preference bias finding?
Models judge their own generated captions as correct far more often than captions from other models. For example, Gemini 3 Flash scores above 40 F1 on other models' captions but drops to 21.9 on its own; pairing models in an ensemble largely removes this blind spot.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (5, 6, 10, 11, 12, 16) were extracted as garbled axis labels; exact per-point values in plots could not be verified, only table numbers.
- Reproduced baselines are marked only with a † legend; which rows are reproduced versus cited is not fully resolvable from the text.
- Metadata (arXiv:2609.21521, 2026 dates) could not be independently verified; this review relies solely on the embedded extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.