Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction
Qi Chen, Yunfei Chu, Yunfei Chu, Haolin He, Yifan Yang, Yifan Yang, Zihan Liu, Yuxuan Wang, et al.
cs.CL · cs.CV · cs.MM · cs.SD
Quick verdict
ODU-Bench makes demand detection and contextual intent inference an explicit benchmark; pipeline is solid, but an LLM-judged metric carries 60% of the score.
Summary
The paper introduces Omni Demand Understanding (ODU), a task and 2,078-scene benchmark (ODU-Bench) testing whether multimodal models can detect whether a user demand exists in an audio or audio-visual interaction and infer intent from multimodal and conversational context. Scenes combine taxonomy-guided agentic video generation (1,801 synthetic) with 277 human-recorded interactions by 30 actors; annotations are reconstructed from realized media via ASR, MLLM observation, and human verification, and a transcript-only discriminator rejects candidates solvable from wording alone. Evaluation spans five dimensions (detection, key-point coverage, localization, transcription, user profile) weighted 0.15/0.60/0.10/0.10/0.05. Among 14 native models, Gemini 3.1 Pro leads audio-visual at 72.6 Avg yet recovers only 44.7% of context-dependent key points versus 82.9% for spoken-request points; Seed 2.0 Lite leads audio-only (77.3). Eleven of 14 models falsely trigger on over 50% of no-demand scenes. In a blinded A/B study (200 scenes), 85.8% of decisive judgments favored responses given demand annotations. Limitations: the primary metric M2 is assigned by a single LLM judge under a lenient criterion; 86.7% of scenes are synthetic; construction and judge prompts are withheld; human-verification agreement rates are unreported.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ODU establishes demand-presence detection plus contextual intent inference as an explicit multimodal benchmark target not jointly covered by prior interaction or intent benchmarks (Table 1).
- Current native MLLMs show a systematic evidence-channel gap: Gemini 3.1 Pro recovers 82.9% of spoken-request key points but only 44.7% of context-dependent ones (gap 38.2 pp, 95% CI [35.4, 41.1]).
- False triggering is prevalent: 11 of 14 models trigger on more than 50% of no-demand scenes, and the best AV model still does so on 40.6%.
- Providing reference demand annotations substantially improves downstream response quality, with 85.8% of decisive blinded judgments favoring the annotated condition (App E.4).
- Benchmark results transfer acceptably from synthetic to human-recorded interactions for leading models (Table 4), supporting the validity of the synthetic construction pipeline.
Strengths
Novel, well-motivated problem formulation
Table 1 shows ODU uniquely combines demand-presence detection, open-ended intent prediction, and visual+acoustic cue inputs with multi-turn dialogue; prior benchmarks (OmniMMI, MultiVox, OmniInteract, Full-Duplex-Bench-v2, MIntRec2.0, GUIDE, EgoIntrospect) each cover only subsets. Making demand understanding an explicit target rather than an implicit precondition of response quality is a genuine conceptual contribution.
Above-average construction rigor for a synthetic benchmark
The transcript-only discriminator (§4.2) rejects positives whose intent is recoverable from the utterance alone and negatives whose no-demand status is textually obvious — a concrete difficulty control. Ground truth is media-grounded (§4.3), reconstructed from ASR plus multi-MLLM observation of the realized media rather than copied from scripts, with key points re-checked against media and human review across all fields.
Quantified judge and coverage audits
Realism check via human-recorded subset with honest negatives
Downstream utility is demonstrated, not assumed
Weaknesses
Primary metric rests on a single lenient LLM judge
86.7% synthetic scenes with a narrow realism check
FTR excluded from the aggregate score
Human verification and A/B study are unquantified
Affiliation overlap across generation, annotation, judging, and evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper formalizes Omni Demand Understanding (ODU): given an audio or audio-visual interaction stream, decide whether the final user turn contains a demand directed at the assistant and reconstruct the intent plus the context required to resolve it (§3). Five prediction targets — demand presence, structured intent with required context, temporal span, transcript, user profile — are scored as M1–M5 with weights 0.15/0.60/0.10/0.10/0.05 (Table 2). The central claims are: (1) existing benchmarks (Table 1: OmniMMI, MultiVox, OmniInteract, Full-Duplex-Bench-v2, MIntRec2.0, GUIDE, EgoIntrospect) do not jointly evaluate demand presence and open-ended multimodal intent; (2) native MLLMs systematically separate perception from contextual inference — Gemini 3.1 Pro recovers 82.9% of spoken-request key points but 44.7% of context-dependent ones; (3) demand understanding is a prerequisite for good responses, evidenced by a blinded A/B study. Claims are appropriately scoped and largely supported by the experiments presented.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ODU-Bench?
ODU-Bench is a public benchmark of 2,078 audio and audio-visual interaction scenes (1,801 synthetic, 277 recorded with human actors) that tests whether AI models can tell when a user is actually making a request to the assistant, and reconstruct the full intent — including information left implicit and inferable only from visuals, non-speech audio, or dialogue history.
What are the benchmark's headline results?
Across 14 native multimodal models, Gemini 3.1 Pro scored highest on audio-visual scenes (72.6/100 aggregate) and Seed 2.0 Lite on audio-only (77.3). Even the best model recovered only 44.7% of information requiring contextual inference (versus 82.9% for explicitly spoken requests), and 11 of 14 models wrongly responded on over half of the scenes containing no real demand.
Is the ODU dataset and evaluation code public?
Yes. The benchmark media and annotations are on Hugging Face (qc316/odubench), with a demo at odubench.github.io and evaluation code plus prediction/judge prompts released. Commercial use is prohibited, and the human-recorded portion is restricted to benchmark evaluation; construction-side prompts are not published.
Why does the 44.7% figure matter?
It isolates the key information a model must infer from visual, acoustic, or conversational context rather than from transcribed speech. The roughly 38-point gap to the 82.9% recovery of spoken-request content shows today's models understand far better what users say than what they mean — the core bottleneck for natural voice assistants.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Heatmap figures (5, 8, 9) extracted as flat number streams; only prose summary statistics from those sections were used.
- Some passages were truncated in extraction (parts of §4.1, E.1 middle, references); appendix structure was reconstructed from the contents listing and surviving text.
- Model names/results (e.g., Gemini 3.1 Pro, GPT-5.4, Seed 2.0 Lite) are reported as stated in the paper and were not independently verifiable from the text.
- Human-recorded subset analyses (Table 4) cover only the Chinese AV slice; full-subset claims would need the released data to verify.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.