LiveProBench: Can Streaming Video Models Really Interact Like Humans?
Kaixuan Du, Xin Wan, Hang Zhang, Meng Cao, Dai Guan, Ming Chen, YuKun Wang
cs.LG
Quick verdict
Solid streaming benchmark for proactive interaction with paired silence scoring; a human-score inconsistency, small subtasks, and no data link temper it.
Summary
The paper introduces LiveProBench, a benchmark that evaluates streaming video models for proactive interaction: a model watches a continuous stream at one-second intervals under a standing request (e.g., 'tell me when the kettle boils') and must decide at every step whether to speak or stay silent, without an explicit response cue. Six subtasks vary timing tolerance and trigger clarity: Event Sensitivity geometrically combines response and silence rates on the same recording; four window-based subtasks classify early, in-window, and missed responses; Duplicate Counting demands exact event counts. The suite spans 5.4 hours and 18,146 evaluated one-second decisions. Six systems are compared, including the authors' Qwen3-Omni-SFT, which leads the audio-disabled primary setting with 51.37 AVG versus a 90.42 human agent (Table 2; §3.2 prose cites 92.99 — an internal inconsistency). Premature responses outnumber missed ones for half the models (factors 2.1–28.2), and no system dominates all subtasks. Limitations: no code or data link, unspecified recording provenance, no significance testing despite small subtasks (DC n=67, WR n=80), and a winning model trained on data whose structure mirrors the benchmark.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing streaming benchmarks are reactive — they query a model at an evaluator-chosen timestamp and therefore never assess whether or when a response is warranted.
- LiveProBench evaluates every one-second stream interval without a response cue, with six subtasks including a paired Event Sensitivity score (geometric mean of response and silence rates on the same recording) that prevents degenerate policies.
- Premature responses outnumber missed responses for half of the evaluated models (factors 2.1–28.2 in the audio-disabled comparison), identifying premature speech as the predominant temporal error mode.
- The best system, the authors' Qwen3-Omni-SFT, scores 51.37 AVG against a human agent ceiling (90.42 per Table 2; 92.99 per §3.2 prose), with deficits concentrated in Sustained Silence and Duplicate Counting.
- Enabling audio shifts response timing in model-dependent directions: it raises Qwen3-Omni-SFT's early responses from 17.7% to 48.4%, while removing audio from ROMA raises its response rate but lowers its silence rate.
Strengths
Paired, content-controlled silence scoring
ES (Eq. 2, §2.4) evaluates response rate and silence rate on the same recording via a geometric mean, controlling the content confound the authors attribute to prior proactive suites; it exposes degenerate policies that averages hide (M4: 3.7% response / 95.5% silence → arithmetic 49.6 masks failure, geometric 18.8 does not, §3.2).
Time-resolved protocol with verifiable internal arithmetic
Subtask decisions sum exactly to the stated 18,146 (1,387+1,992+2,530+1,977+9,384+876); window examples 218+80+238+92 match Fig. 2's captioned 628; SS's 92 examples at ~102 decisions reproduce 9,384 (§2.2).
Human ceiling grounds task difficulty
Error decomposition invisible to timestamp-based protocols
Weaknesses
Headline human-score inconsistency
No uncertainty quantification
Data provenance and release unstated
Benchmark circularity with the winning model
Heterogeneous silence mechanisms inside one ranking
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
LiveProBench (arXiv:2609.12658v2, cs.LG) addresses a real evaluation gap: streaming benchmarks such as OVO-Bench, StreamingBench, and OVBench query a model at an evaluator-chosen timestamp, so whether and when a model should answer is never measured. The paper claims three contributions: (1) a per-second, cue-free protocol preserving each model's native silence mechanism; (2) six subtasks crossing timing tolerance (zero, fixed 3 s, event-duration) with trigger clarity (single decisive second, underway activities, generic anomaly request); (3) the empirical finding that premature responses dominate temporal errors for half the evaluated systems. The design claims are well supported and internally consistent; the headline human-gap claim carries a numerical inconsistency, and reproducibility signals are weak.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LiveProBench?
LiveProBench is a benchmark that tests whether streaming video models know when to speak. Under a standing request (for example, 'tell me when the kettle boils'), a model watches a video one second at a time and must decide at every step whether to respond or stay silent, with no explicit cue telling it when to answer. It has six subtasks covering response timing, sustained silence, anomaly alerts, and counting repeated events.
What did LiveProBench find about current video models?
Across six systems, the best model (Qwen3-Omni-SFT, audio disabled) averaged 51.37 points versus 90.42 for a human agent (Table 2; the paper's prose states 92.99 for the human). For about half the models, premature responses were 2.1 to 28.2 times more frequent than missed events, showing that deciding when to respond — not just recognizing events — is a major unsolved gap.
How is the Event Sensitivity (ES) subtask scored?
ES evaluates both reacting and restraining on the same video: it takes the rate of responding at the annotated alert time and the rate of staying silent through the surrounding segments, then combines them with a geometric mean. A model must do both well; one that talks constantly or never speaks scores poorly even if a plain average would look acceptable.
Is the LiveProBench data or code publicly available?
The paper describes released annotations (source recording, event labels, standing requests, reference responses) but gives no repository link, license, or code-release statement, and it never identifies where the video recordings came from, so availability cannot be confirmed from the paper text alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Fig. 2's per-model early/in-window/missed bars are extracted with interleaved labels; only values restated in prose (factors 2.1–28.2; Qwen3-Omni-SFT 17.7/57.6/24.7; ROMA 74.2 vs 18.0) are cited verbatim, and M4's 94.4% early share is read from the figure block with hedging.
- Table 2 is extracted with whitespace-collapsed columns; row-to-column mapping was cross-verified against the paper's stated geometric means (e.g., ROMA 49.5×48.0→48.7; Qwen3-Omni-SFT 41.7×84.6→59.4), which all reproduce, but a single-cell misread cannot be fully excluded.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.