Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation
Jintao Zhang, Kai Jiang, Jintao Chen, Xu Wang, Deyuan Liu, Jungang Li, Dechuang Chen, Ming Lin, et al.
cs.CV · cs.LG
Quick verdict
Vidu S2 claims real-time 720p interactive avatars and streaming video editing; broad benchmark wins, but no ablations, significance tests, or code.
Summary
Vidu S2 is a real-time video generation system from Tsinghua University and Shengshu Technology with two components: S2-Avatar, an audio-driven interactive digital character supporting 720p at 25-42 FPS with mid-stream updatable references and dancing-level instruction following, and S2-Editing, which applies style transfer, virtual try-on, subject replacement, and background replacement to live video streams. The core training contribution is Self-Replay Forcing (SRF): a detached autoregressive rollout is re-noised following Diffusion Forcing and replayed in a gradient-enabled causal pass, letting gradients cross block boundaries without backpropagating through the rollout. A one-step super-resolution refiner with asymmetric noise-level caches lifts 540p latents to 720p, and an inference stack (hybrid Sage/Sparge/SLA attention, per-block W8A8 GEMM, CUDA graphs, Ulysses context parallelism) sustains real-time throughput. On StreamAV-Bench the avatar leads all nine reported metrics; the editor tops Sparkle-Bench (3.74 overall), joint OpenVE-RefVIE (4.26), and ViViD try-on (VFIDI 9.95 vs 19.51 for CatV2TON). Internal GSB studies favor it over Runway, PixVerse, HeyGen, XMax-X2.0, and Decart-Lucy2.5. Key limitations: no ablations isolating SRF or the refiner, avatar GSB comparisons with apparently about 7 judgments per criterion, no published latency/cost figures, closed models and data, and purely qualitative spatial-video evaluation.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Self-Replay Forcing improves autoregressive distillation by re-noising the student's detached rollout and replaying it with cross-block gradient flow (Section 2.2, Eq. 4).
- Vidu S2-Avatar generates 720p audio-driven avatars in real time at 25-42 FPS, with references updatable at any stream moment and dancing-level instruction following.
- Vidu S2-Editing performs real-time streaming editing across style transfer, virtual try-on, subject replacement, and background replacement, outperforming offline and streaming baselines on public benchmarks.
- Real-time monocular-to-stereo conversion and streaming stereo editing are feasible within the same pipeline, demonstrated qualitatively for VR headsets (Section 4, Figure 13).
Strengths
SRF is a well-specified, well-motivated training contribution
Section 2.2 (Eq. 4) precisely targets two stated limits of Self-Forcing — clean detached history and no cross-block gradient — by re-noising a detached rollout per Diffusion Forcing and replaying it in one gradient-enabled causal pass with DMD plus perceptual loss.
Full-stack real-time engineering with concrete components
Sections 2.3/3.3 name specific mechanisms: layer-wise hybrid Sage/Sparge/SLA attention, per-block W8A8 GEMM, Triton/CUDA kernel fusion, CUDA Graphs, quantized Ulysses context parallelism, and cross-module GPU timeline sharing (VAE encoder/backbone/Refiner/decoder).
Broad, protocol-conscious evaluation design
Thoughtful data engineering
First-place results across all reported public benchmark columns
Weaknesses
No ablation studies at all
Very small human-evaluation samples for the avatar comparisons
Low reproducibility
Spatial video claims are purely qualitative
Judge-based headline metrics without bias controls
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Vidu S2 (Tsinghua University and Shengshu Technology, arXiv 2609.11638) is a system paper covering two real-time models - **Vidu S2-Avatar** (audio-driven interactive digital characters) and **Vidu S2-Editing** (streaming video editing) - plus a feasibility exploration of real-time stereo ('spatial') video and a serving stack. Central claims: (1) Self-Replay Forcing (SRF) improves autoregressive distillation over Self-Forcing; (2) real-time 720p generation at 25-42 FPS with mid-stream reference updates and dancing-level instruction following; (3) real-time editing across four tasks; (4) 'Vidu S2 outperforms all baselines' (Abstract, Contribution 5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Vidu S2?
Vidu S2 is a real-time video generation system from Tsinghua University and Shengshu Technology with two models: an interactive audio-driven digital avatar (720p at 25-42 FPS, reference images replaceable mid-stream) and a real-time video editor for live streams (style transfer, virtual try-on, character and background replacement), plus a pipeline converting streams into stereo 'spatial video' for VR.
What is Self-Replay Forcing?
It is the paper's training method: the model first generates a long autoregressive rollout with detached gradients, that self-generated trajectory is re-noised per segment following Diffusion Forcing, and it is replayed in one gradient-enabled causal pass, so later segments can improve earlier ones without backpropagating through the original rollout.
Is Vidu S2 code or model weights available?
No. The paper only points to an online demo at vidu.com/vidu-stream. Training data, internal benchmarks, and prompts are proprietary, so the reported results cannot currently be reproduced independently from the paper.
How does Vidu S2 compare to competitors?
The paper reports first place in every column of its public benchmark tables (StreamAV-Bench, Sparkle-Bench, joint OpenVE-RefVIE at 4.26, ViViD VFIDI at 9.95) and favorable internal human comparisons against Runway, PixVerse, HeyGen, XMax-X2.0, and Decart-Lucy2.5 - though some avatar human comparisons involve only about 7 judgments per criterion and no code or weights are released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 5-13 are images; only their extracted caption/overlay text is available, so qualitative comparison claims and Figure 5's exact ratings could not be visually verified.
- Per-bar denominators in Figure 6 are not stated numerically; the ~7-judgments figure is an inference from the 1/7 percentage granularity in the extracted text, not an explicit n reported by the paper.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.