PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation
Yuchen Sun, Qian Yang, Jun Wang, Detai Xin, Guoqiao Yu, Guanglu Wan, Qi Jia
cs.MM · cs.AI · cs.SD
Quick verdict
Claims first audio-centric T2AV benchmark (900 samples, MLLM judge); thorough validation but data withheld and top models lack human-alignment checks.
Summary
PRISM-Bench is a benchmark for evaluating text-to-audio-video (T2AV) generation with audio as the primary focus. The authors curate 900 human-verified audio-video clips (300 each for On-screen, Off-screen, and Mixed sound-source visibility) from web and licensed sources, annotate them via a Gemini 2.5 Pro / Qwen3-Omni / GPT-5 fusion pipeline with human correction, and factorize evaluation along audio type (Speech, Music, Sound) crossed with visibility. Four dimensions (Audio-Visual Coherence, Audio Quality, Audio Expressiveness, Prompt Following) are scored by 35 type-aware criteria through a blind, side-by-side MLLM-as-a-Judge protocol (Gemini-3.1-pro-preview, temperature 0) comparing each generated clip against its ground-truth reference in two stages. Evaluating eight systems (Seedance 2.0, Kling v3 Omni, Veo 3.0/3.1, Sora 2, LTX-2, Ovi, MOVA), Seedance 2.0 leads all subsets (Final 97.73 On-screen, 78.11 Off-screen, 99.88 Mixed), yet its AV Coherence (20.91) still trails ground truth (20.93), supporting the paper's finding that fidelity outpaces visible-source grounding. Human agreement exceeds 70% on average, but GT-vs-model alignment drops to 68% for Sora 2 and 63.4% for the close LTX-2-vs-Ovi pairing. Key limitations: the 900 clips are withheld for licensing reasons, the judge is a proprietary API from the same family as the annotation models, human validation excludes the two top-scoring systems, and no significance testing backs the rankings.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PRISM-Bench is the first benchmark to jointly stratify T2AV audio evaluation by audio type and sound-source visibility, built on 900 human-verified clips with 35 criteria across four perceptual dimensions.
- An enhanced blind, side-by-side MLLM-as-a-Judge protocol (Gemini-3.1-pro-preview, temperature 0) achieves over 70% mean agreement with human raters.
- Frontier proprietary systems, led by Seedance 2.0 (Final 97.73/78.11/99.88 on On-screen/Off-screen/Mixed), hold at least a 35-point lead over the best open-source models on every subset.
- Current T2AV paradigms overfit perceptual fidelity: even the leader's AV Coherence stays at or below ground truth (20.91 vs 20.93 On-screen) while Audio Quality and Prompt Following exceed the reference.
Strengths
Genuinely diagnostic crossed taxonomy
Stratifying audio type (Speech/Music/Sound) by visibility (On-screen/Off-screen/Mixed) exposes failures aggregate scores hide: Seedance 2.0's On-screen AV Coherence total is 20.91 vs 25.24 Audio Quality despite a 97.73 Final (§4.2), and Kling-v3-omni's On-screen Music AV Coherence drops to 4.06 vs 7.75 for Speech.
Unusually careful judge validation
Appendix A separates ground-truth contextual drift (d_bias 0.152–0.281, Tables 2–3) from inter-model signal (d_signal 0.521–0.814, SNR 1.85–4.21), localizing instability in sync-sensitive AV branches (2.145 speech, 2.314 music); Table 4's judge ablation rejects the most stable judge (Qwen3-Omni, SNR 1.11) for compressing differences, articulating a stability-vs-discriminability trade-off.
Transparent, auditable protocol
Candid reporting of protocol non-idealities
Timely, broad system coverage with documented configs
Weaknesses
Benchmark data withheld, core results unverifiable
Judge–annotation circularity unexamined
Human validation excludes the top two systems
No statistical inference behind rankings
Heterogeneous inference settings confound comparisons
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PRISM-Bench (arXiv:2609.04867, ACM MM '26) proposes the first audio-centric *diagnostic* benchmark for text-to-audio-video (T2AV) generation. Its differentiating move is a crossed taxonomy: audio type (Speech/Music/Sound) × sound-source visibility (On-screen/Off-screen/Mixed), instantiated as three 300-clip subsets totaling 900 human-verified clips (§3.2). Four dimensions — Audio-Visual Coherence, Audio Quality, Audio Expressiveness, Prompt Following — are operationalized through 35 type-aware criteria (§3.3). The central empirical claims: (1) a blind, side-by-side MLLM judge reaches over 70% mean human agreement; (2) frontier proprietary systems led by Seedance 2.0 dominate open-source models by at least 35 Final-score points on every subset; (3) current paradigms overfit perceptual fidelity while visible-source grounding lags — Seedance 2.0's On-screen AV Coherence (20.91) sits just below ground truth (20.93) even as its Audio Quality (25.24 vs GT 23.10) and Prompt Following (25.89 vs GT 24.81) exceed the reference.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PRISM-Bench?
PRISM-Bench is a benchmark for evaluating text-to-audio-video generation models with audio as the main focus. It uses 900 human-verified clips organized by audio type (speech, music, sound) and whether the sound source is visible on screen, and scores systems on audio-visual coherence, audio quality, audio expressiveness, and prompt following. An AI judge (Gemini) compares each generated clip against a real reference clip in a blind, side-by-side protocol.
Which model performs best on PRISM-Bench?
Seedance 2.0 ranks first on all three subsets with calibrated Final scores of 97.73 (On-screen), 78.11 (Off-screen), and 99.88 (Mixed), at least 35 points ahead of the strongest open-source models (LTX-2 and Ovi). However, even Seedance stays marginally below the real-reference score on audio-visual coherence for on-screen audio (20.91 vs 20.93).
Is the PRISM-Bench dataset publicly available?
No. The 900 clips are kept internal because of licensing and redistribution constraints; only a public leaderboard and the evaluation code are released. This means outside teams cannot re-run the benchmark or independently verify reported scores.
How reliable is the AI judge used by PRISM-Bench?
The authors report mean human-judge agreement above 70% across dimensions and pairwise comparisons. Reliability is lower in hard cases: 68% agreement for Sora 2 against ground truth and 63.4% for the close LTX-2-vs-Ovi pairing. Human validation covered only outputs from Sora 2, LTX-2, and Ovi — the top two systems (Seedance 2.0, Kling v3 Omni) were not human-checked.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's radar values and Figure 2(c)'s word cloud are extracted only as labels/layout text, so per-spoke calibrated scores in the overview figure could not be cross-checked against the narrative.
- A few Table 1 cells show OCR spacing artifacts (e.g., Kling Off-screen Speech Prompt Following appears as '26 54'; the narrative confirms 26.54); all numbers cited in this review were legible and verified against the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.