From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models
Meng Luo, Yicheng Liu, Jiahao Wang, Yuanxing Zhang, Xin Tao, Pengfei Wan, Kun Gai, Hao Fei
cs.CV · cs.AI
Quick verdict
A 9-dimension video-reasoning benchmark plus a text-reward prompt rewriter with solid gains, but the 380-sample eval is judge-circular and statistically thin.
Summary
The paper introduces VWG-Bench, a 9-dimension, 38-task benchmark (380 samples, 10 per task) for probing 'think-with-video' reasoning in image-to-video generators, evaluated by a three-level VLM-as-Judge protocol — video quality, progress consistency and implicit-rule adherence, and progress/last-frame goal realization — each scored 1–5 by Gemini-3.0-Pro against automatically generated annotations. Baselines show strong rendering but weak reasoning: Veo3.1 leads at 3.55 overall while Wan2.2 scores 2.13 with rule-following at just 1.56. The proposed Vid-PRE, built on Qwen3-VL-8B, is first SFT-trained on ~20K rejection-sampled (image, prompt, CoT, prompt) tuples generated by Gemini-3.0-Pro, then GRPO-tuned on 1.5K hard prompts with purely text-based rewards (intent, image consistency, solution correctness, format). Gains are substantial: Wan2.2 improves from 1.72 to 2.58 on MME-CoF, from 26.65 to 44.48 on V-ReasonBench (spatial cognition 1.32→36.25), and from 2.13 to 2.51 on VWG-Bench, also beating the built-in Wan prompt extension; human–judge agreement is ρ 0.69–0.80 over 200 videos. Main limitations: no significance tests or variance on the 380-sample benchmark; the same proprietary model handles annotation, data generation, filtering, rewards, and final judgment (circularity); reward weights are unspecified; supplementary per-dimension numbers are hard to reconcile with headline tables; and no zero-shot LLM rewriter baseline isolates the training benefit.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Current video generators render well but fail logic-heavy, rule-constrained tasks: on VWG-Bench even Veo3.1 reaches only 3.55/5 overall, and Wan2.2 scores 1.56 on implicit rule-following (Table 2).
- A three-level VLM-as-Judge protocol (video quality, task-level consistency/rules, sample-level goals) diagnoses failure modes that holistic scoring masks, and agrees with human experts at Spearman ρ 0.69–0.80 (Table 4).
- Vid-PRE, trained with SFT then GRPO on purely text-based rewards, substantially improves reasoning across generators without architectural change (Wan2.2: VWG-Bench 2.13→2.51, MME-CoF 1.72→2.58, V-ReasonBench 26.65→44.48), beating the built-in Wan prompt extension.
- Text-based rewards are the right design for prompt-reasoner RL because they are stable, cheap, and generator-agnostic, with solution correctness the dominant reward component (supplementary reward ablation).
Strengths
Structured three-level protocol decouples rendering from reasoning
Unlike the single-level scoring of the eight prior benchmarks compared in Table 1, VWG-Bench scores video quality, task-level progress consistency/implicit-rule adherence, and sample-level progress/last-frame goals independently on 1–5 scales; the full judge and annotation prompt templates are printed in Appendix C, a real transparency plus.
Quantified rendering–reasoning gap across six generators
Table 2 shows the pattern consistently: Wan2.2 scores 2.32 video quality vs 1.56 rule-following; Sora2 drops from 3.45 quality to 2.98 last-frame goal; best model Veo3.1 reaches only 3.55 overall on a 5-point scale, supporting the difficulty claim.
Judge reliability is empirically validated, not just asserted
Ablations actually isolate where gains come from
Deployment cost is documented
Weaknesses
No statistical significance or variance reporting
End-to-end judge circularity
Missing zero-shot rewriter baseline
Supplementary per-dimension numbers do not reconcile with headline tables
Generalization tested only within the Wan family
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: VWG-Bench and Vid-PRE
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is VWG-Bench?
VWG-Bench is a benchmark introduced in this paper to test whether video generation models can 'think with video' — execute symbolic rules, respect physics, and reach goals. It spans 9 reasoning dimensions and 38 tasks with 380 image–prompt samples, and scores generated videos 1–5 at three levels: video quality, task-level rule adherence, and sample-level goal realization, using a VLM (Gemini-3.0-Pro) as judge.
What is Vid-PRE and how much does it improve results?
Vid-PRE is a prompt-rewriting model (built on Qwen3-VL-8B) that reasons about the task and rewrites user prompts into explicit, constraint-aware instructions for image-to-video generators. In the paper it lifts Wan2.2 from 1.72 to 2.58 on MME-CoF, from 26.65 to 44.48 on V-ReasonBench, and from 2.13 to 2.51 on VWG-Bench, also beating the built-in Wan prompt extension; rewriting adds only 3–5 seconds (<8% latency).
Which video models were evaluated and which scored best?
Six generators were tested on VWG-Bench: Wan2.2 (2.13 overall), Wan2.5 (2.75), Wan2.6-Flash (2.80), Kling-2.5-Turbo-Pro (2.97), Sora2 (3.44), and Veo3.1 (3.55, the best). All scored much higher on visual quality than on rule-following and goal completion; the supplement adds Seedance 2.0 at 3.52 versus Veo3.1's 3.55.
Is the VWG-Bench dataset and code publicly available?
The paper states that data and code are publicly available via the Hugging Face dataset KlingTeam/VWG-Bench. Note that the evaluation judge and training-data generator is the proprietary Gemini-3.0-Pro, so fully reproducing the reported numbers depends on access to that model.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are embedded images not included in the extracted text, so the taxonomy diagram, training framework, and the qualitative video comparisons of Figure 3 could not be visually verified.
- Sub/superscripts are flattened by extraction (e.g., 'Vid-PRESFT' for Vid-PRE_SFT in Table 5); all tables were reconstructed from flattened text, so cell-to-column mapping relied on layout cues.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.