TempCloze: Can Video-LLMs Identify the Missing Middle?
Wenqi Pei, Henry Hengyuan Zhao, Yilai Liu, Jiahao Meng, Han Chen, Ziyu Wang, Hongyang Du
cs.CV · cs.AI
Quick verdict
Video-cloze benchmark (1,521 videos, 31 Video-LLMs): alignment, not semantics, is the bottleneck; solid shortcut controls but no significance tests.
Summary
TempCloze is a video-cloze benchmark probing visual temporal reasoning in Video-LLMs without language-mediated questions. Given a video's beginning and ending clips, a model must select the true missing middle from four same-source candidates. The 1,521 videos (12–90 s, from seven sources such as LVD-2M, EgoLife, and MiraData) pass duration, GPT-o3 caption-suitability, quality, and optical-flow filtering. Distractors instantiate three diagnostic dimensions: Semantic (wrong event), Alignment (shifted or expanded span), and Progression (Reversed/Reordered/Repeated), sharing scenes and objects to suppress appearance cues. Evaluating 10 proprietary and 21 open-source Video-LLMs, the authors find Alignment is the bottleneck: proprietary models average 48.13% on Alignment versus 70.73% Semantic and 67.72% Progression (open-source: 26.54% versus 34.00%/36.97%), while humans reach 97% and the best model (Seed1.8 with thinking) 88.58% mean. Error analysis shows Expanded dominates Alignment failures and Reversed dominates Progression failures. Behavioral analyses on the Mixed (300) and Hard (150) subsets reveal candidate-order instability (clip flip rates 32–61%), stronger reliance on beginning than ending context, degradation as visible span or frame density grows, and test-time-scaling gains that preserve the dimension ordering. Limitations include absence of statistical significance testing, a 100-video human baseline, use of GPT-o3/Gemini-2.5-Pro in construction while their vendors' models are evaluated, and possible over-penalization of Expanded clips that contain the true span.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Temporal Alignment — placing the missing clip precisely between endpoints — is the primary bottleneck: proprietary models average 48.13% on Alignment versus 70.73% Semantic and 67.72% Progression.
- Same-source distractors with formally defined temporal edits suppress appearance and language shortcuts, as validated by near-uniform edge baselines (25.49–26.35% vs 25%) and zero boundary-frame leakage.
- Models rely more on beginning than ending context, and longer visible spans or denser sampling hurt, with Alignment degrading most.
- Accuracy is stable but choices are not: candidate-order permutations flip selected clips in 32.4–60.7% of instances while mean accuracy moves under 4 points.
- Test-time scaling (pass@k up to 5, roughly +30 overall points for Gemini2.5-Pro and Qwen3.5-397B-A17B) never changes the dimension ordering, keeping Alignment hardest.
Strengths
Exemplary shortcut controls
Zero exact boundary-frame matches across 18,252 candidate clips (Table 5), DINOv2 edge baseline at 25.49–26.35% against the 25% uniform floor (Table 7), and a single-frame ablation collapsing Seed1.8 from 83.43% to 17.31% mean (Table 8) together demonstrate the task demands temporal integration rather than appearance or boundary matching.
Formalized, reproducible distractor algebra
Appendix A fixes exact edits (shifts δ=ℓ/2 within [0.1T,0.2T]; symmetric expansion; reverse/reorder/repeat), construction and evaluation prompts are printed verbatim (Appendix G), and per-stage filtering statistics are tabulated (Table 9, e.g., LVD-2M 82,196→192 retained).
Breadth of evaluation with a conservative human ceiling
Diagnostics beyond headline accuracy
Weaknesses
No inferential statistics anywhere
Expanded distractor confounds the headline claim
Construction/evaluation vendor overlap
Uniform frame budget may depress open-source scores
Failure-selected Hard subset
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
TempCloze (arXiv 2609.01515) is a benchmark paper proposing a video-cloze evaluation for Video-LLMs: given the beginning and ending clips of a video, the model must identify the true missing middle from four candidates drawn from the same source video (§3.1). The paper claims: (1) existing temporal-reasoning benchmarks are language-mediated and admit linguistic shortcuts (§1, citing text-only LMs exceeding random baselines by over 25%); (2) same-source, controlled-edit candidates suppress appearance and language priors; (3) temporal Alignment (when) — not Semantic content (what) or Progression (how) — is the primary bottleneck across 31 evaluated Video-LLMs. Auxiliary claims concern structured error signatures (Expanded, Reversed), candidate-order instability, and beginning-over-ending context reliance.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the TempCloze benchmark?
TempCloze is a video cloze benchmark for Video-LLMs: a model sees the beginning and ending clips of a video and must pick the true missing middle from four candidates drawn from the same video. Distractors are designed along three dimensions — Semantic (wrong event), Alignment (wrong timing), and Progression (wrong unfolding) — using 1,521 filtered videos from seven public sources.
What is the main finding of TempCloze?
Across 31 evaluated Video-LLMs, temporal Alignment — judging when an event should occur between observed endpoints — is the weakest capability. Proprietary models average 48.13% on Alignment versus about 68–71% on Semantic and Progression, while humans score 97%. The most common Alignment failure is picking the Expanded clip, which contains the correct event but the wrong time span.
How good are current Video-LLMs on TempCloze?
The best model (Seed1.8 with thinking) reaches 88.58% mean dimension accuracy and solves all three dimensions on 70.81% of videos. Open-source models average 32.51% mean accuracy, and the weakest proprietary model (Grok4.1) sits at 24.11%, essentially at the 25% chance level.
Is the TempCloze code and data available?
Yes, the paper links a GitHub repository (github.com/CedricPei/Temporal-Cloze). Videos come from seven public datasets, and both the GPT-o3 suitability-filter prompt and the evaluation prompt are printed in the appendix.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Numeric grids embedded in Figures 2, 3, and 5–9 extract as unlabeled number sequences, so cell-level figure values beyond those restated in the prose could not be independently verified; all numbers cited in this review come from prose or labeled tables.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.