MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant
Yi-Jen Shih, Shih-Yun Shan Kuan, Guan-Ting Lin, Kai-Wei Chang, Siddhant Arora, Shu-wen Yang, Abdelrahman Mohamed, Shinji Watanabe, et al.
eess.AS · cs.AI · cs.CL
Quick verdict
First audio benchmark for multiparty voice-agent participation; solid construction and ablations, but fully synthetic clean audio limits ecological validity.
Summary
MP-Bench is presented as the first benchmark evaluating conversational voice agents as active participants in multiparty (four-party) spoken conversations, addressing a gap left by dyadic-interaction and passive-comprehension benchmarks. The authors build 927 tasks from two LLM-generated, ElevenLabs-synthesized scenarios: a Discussion scenario (the agent is invited to adjudicate a three-way dispute) and a Turn-based Game scenario (the agent must speak only on its cross-team turn). Tasks test understanding (QA with speaker attribution), turn-taking (explicit, implicit, negative), and response appropriateness; outputs are transcribed by streaming ASR, with onsets from word timestamps and judgments by GPT-5. Across 12 voice agents plus text-LLM references, realtime agents score at or below 33% on multiparty comprehension (Covo-Audio best at 32.6% overall accuracy) and sit near the 50% chance baseline on implicit/negative turn-taking, while a Gemini-3.1-Pro reference with oracle speaker-labeled transcripts reaches 98.4% turn-taking accuracy, attributing the deficit to audio-based speaker attribution. Ablations (single-speaker game variant, similar-voice subset, multiparty-aware prompts) and human/judge agreement checks (Krippendorff's α=0.745; κ=0.646) support the findings. Main limitations: fully synthetic, clean, non-overlapping audio; a single decision point per task; the DuplexCascade row in the main table is never described or cited; and no confidence intervals on headline comparisons.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MP-Bench is the first benchmark to evaluate voice agents as active participants in multiparty spoken conversations, spanning understanding, turn-taking, and response appropriateness.
- Realtime voice agents score at or below 33% on multiparty comprehension and remain near the 50% chance baseline on implicit and negative turn-taking.
- The turn-taking failure stems from recovering speaker attribution from audio: ASR-only text input yields chance-level turn-taking (49.22%) while oracle speaker-labeled transcripts reach 98.44%.
- These failures reflect inherent multiparty complexity, not weak instruction following (single-speaker ablation), and persist under multiparty-aware system prompts.
Strengths
Genuine gap, well positioned
Sec. 2 documents that existing multiparty benchmarks test passive understanding and concurrent turn-taking work is text-only; MP-Bench forces joint acoustic-plus-contextual processing directly from audio.
Audio-necessity by construction
Dialogues and QA pairs are retained only when a text LLM fails without speaker labels but solves with anonymized labels (Sec. 4.2.3, 4.3.1), and all tasks are human-verified — tasks cannot be trivially transcript-solved by design.
Oracle references localize the failure
Unusually thorough validity checks
Quantified domain-gap honesty
Weaknesses
Undocumented system in the main table
No uncertainty on headline numbers
Degenerate and lenient metric choices
Channel asymmetry between model classes
Ecological validity of fully synthetic audio
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MP-Bench (arXiv 2609.13076) claims to be the first benchmark evaluating conversational voice agents as *active participants* in multiparty spoken conversation, rather than in dyadic exchanges or as passive listeners. Sec. 2 positions it against passive multiparty benchmarks (Inoue et al. 2025; MSU-Bench; M3-SLU) and text-only concurrent turn-taking work (Bhagtani et al. 2026; Nama et al. 2026). The benchmark contains 927 human-verified tasks built from two LLM-authored, ElevenLabs-synthesized scenarios: **Discussion**, where the agent is explicitly invited to adjudicate a three-way dispute, and **Turn-based Game**, where the agent may speak only when its cross-team rule triggers its turn. Central claims: (1) realtime agents score at or below 33% on multiparty comprehension (best: Covo-Audio, 32.60% overall accuracy, Table 3); (2) every realtime system sits within ±3 points of the 50% chance baseline on implicit/negative turn-taking; (3) the deficit lies in speaker attribution from audio, evidenced by ASR-only vs. SpkD-ASR text references (49.22% → 98.44% turn-taking); (4) failures survive multiparty-aware prompting. Note: one abstract version circulating says 22% on comprehension; the embedded abstract and Table 3 support 33%, which I follow.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MP-Bench?
MP-Bench is a benchmark of 927 tasks testing whether voice AI agents can participate in four-person spoken conversations: answering questions about a multi-speaker discussion, knowing when it is their turn to speak, and giving appropriate responses. Dialogues were written by an LLM and spoken with 12 distinct ElevenLabs synthetic voices.
How well do current voice agents handle multiparty conversations according to MP-Bench?
Poorly. Across 12 evaluated voice agents, realtime systems scored at most 32.6% on multiparty comprehension questions and hovered near the 50% chance level on deciding when to speak in implicit or negative turn-taking situations. Only a text model given speaker-labeled transcripts (98.4% turn-taking accuracy) did well, showing the bottleneck is identifying who is speaking from audio alone.
Does MP-Bench use real human recordings?
No. All conversations are synthetic: LLM-written dialogues rendered with ElevenLabs voices. The paper compares audio statistics against the real AMI meeting corpus and notes its audio has no overlapping speech, noise, or reverberation, making it a controlled but simplified setting that likely overstates how easy real multiparty participation is.
Is MP-Bench's code and data available?
The paper links a public GitHub repository (github.com/atosystem/MP-Bench) and prints all generation and judging prompts in its appendix. Full replication still depends on paid services such as ElevenLabs synthesis and GPT-5 as the judge, and one main-table system (DuplexCascade) is not documented anywhere in the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The harness-provided metadata abstract states realtime agents stay 'at or below 22%' on comprehension, while the embedded full-text abstract and Table 3 (best realtime 32.60%) say 33%; this review follows the embedded text.
- Table 3's header row is flattened in extraction; the column mapping was verified against in-text numbers (e.g., ASR-only 49.22 Turn-based TT quoted in Sec. 6.1) and is consistent.
- Figure 2's pipeline labels are jumbled in extraction; the task counts cited (256 dialogue instances, 543 understanding, 384 behavior tasks) were cross-checked against Sec. 3 and sum correctly to 927.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.