MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents
Chenxu Xiong, Dongming Shen, Yuzhi Tang, Wentao Ma, Mu Li, Alex Smola
cs.CL
Quick verdict
Benchmark of 1,152 bilingual multi-speaker voice scenes; careful build and judge validation, but synthetic audio and LLM rubrics cap ecological validity.
Summary
MSI-Bench is a bilingual benchmark (576 English and 576 Mandarin cases) for evaluating whether voice AI agents act correctly in multi-speaker settings, where the right response depends on who said what, who is authorized, and what must be withheld from whom. Six patterns span three capability families — multi-speaker memory, instruction following, and reasoning — each staged as a scripted two-to-three-speaker audio scene with participant context, a function catalog including decoys, and atomic rubrics. Cases come from an LLM generation pipeline and cloned-voice TTS synthesis; 1,420 candidates were manually filtered to 1,152. Twelve models under 15 configurations are scored on all-pass rate, atomic rubric score, tool execution, bystander-interference robustness, and premature response rate, with a DeepSeek V4 Pro judge validated against human labels (κ=0.63 versus 0.56 between annotators). The best hosted configurations pass all rubrics on 66.8% of English cases (Gemini 3.1 Pro) and 54.5% of Mandarin cases (GPT Realtime 2.1 xhigh); the best open-weight configuration (Qwen3-Omni-30B) reaches 34.0% and 19.3%. Transcript ablations indicate open-weight failures are mostly perceptual, while frontier models' residual failures persist on clean transcripts. Main limitations: fully synthetic scripted audio, LLM-generated rubrics judged by an LLM, small 96-case per-pattern cells, and an unexplained uniform Mandarin deficit.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MSI-Bench instantiates 1,152 bilingual, rubric-scored audio scenes (576 English, 576 Mandarin) targeting speaker-scoped memory, instruction following, and reasoning in shared voice interaction.
- No evaluated system is reliable: the best hosted configuration passes all rubrics on 66.8% of English cases (Gemini 3.1 Pro) and 54.5% of Mandarin cases (GPT Realtime 2.1 xhigh), while the best open-weight configuration (Qwen3-Omni-30B) reaches only 34.0% and 19.3%.
- Open-weight failures are chiefly perceptual — speaker-labeled transcripts lift Gemma/MiMo configurations by 21–44 APR points — whereas frontier residual failures persist as speaker-scoped reasoning errors on clean transcripts.
- Conversational restraint is largely independent of answer quality: Gemini 3.5 Flash answers prematurely on 48.2% of probes despite ranking second on APR, and every open-weight configuration answers on at least 66.1% of probes.
- As background speech becomes harder to hear, models miss more task facts while admitting fewer misses (Gemini 3.1 Pro capture falls 62.5%→15.6% from +8 to −8 dB while its miss-admission drops 44.4%→6.2%), compounding into unsupported actions.
Strengths
Clear differentiation from prior voice benchmarks
Section 2 positions MSI-Bench against the Full-Duplex-Bench line, TPI-Bench, WearVox, MSU-Bench and M3-SLU: those evaluate when to speak or speaker-attributed QA, whereas MSI-Bench scores the speaker-grounded what-to-do — answer, refusal, clarification or tool call — under memory, instruction-following and reasoning constraints.
Multi-gate data construction with human filtering
Section 3.2: structural gates (roster vs sampled speaker count, gold tool calls type-checked against declared schemas), a semantic verifier gating answer leakage and gold uniqueness, an ASR/energy gate on every clip, and manual review that kept 1,152 of 1,420 candidates (an 18.9% rejection rate).
Diagnostic ablations cleanly attribute failures
Judge validated beyond typical practice
Conversational restraint measured as a distinct axis
Weaknesses
Fully synthetic, scripted stimuli
Rubric generation and judging are both LLM-based
Small per-pattern samples without uncertainty
Unexplained Mandarin deficit with audio-side confounds
BIR protocol asymmetry across model families
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MSI-Bench (Boson AI; arXiv:2609.24812) targets a gap the authors argue turn-taking benchmarks leave open: in shared voice conversations — meetings, homes, team channels — the correct next assistant action depends on *who* spoke, their authority, and applicable privacy boundaries. The benchmark delivers 1,152 cases (576 English / 576 Mandarin), authored natively per language, spanning six patterns in three families: multi-speaker memory (background speech retrieval, scope tracking), multi-speaker instruction following (selective disclosure, speaker authority constraint), and multi-speaker reasoning (sequential constraint integration, constraint prioritization), across eight Barker-inspired behavior settings. Each case is a 2–3-speaker, 6–12-turn scripted audio scene ending in an assistant-directed handoff, scored by 2–4 atomic rubrics plus a deterministic tool validator. Central claims: (1) no current system is reliable — best hosted APR 66.8% English / 54.5% Mandarin, best open-weight 34.0% / 19.3%; (2) open-weight failures are perceptual while frontier failures are speaker-scoped reasoning; (3) conversational restraint is an independent failure axis.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MSI-Bench?
MSI-Bench is a public benchmark of 1,152 short multi-speaker audio scenes — half English, half Mandarin — that tests whether voice AI agents act on the right person's information, permissions, and constraints. Each case is scored by atomic rubrics covering the answer text and any tool calls, rather than only turn-taking behavior.
How well do current models perform on MSI-Bench?
Poorly in absolute terms. Gemini 3.1 Pro tops the English split with 66.8% of cases passing all rubrics; GPT Realtime 2.1 (xhigh) tops Mandarin at 54.5%. The best open-weight model, Qwen3-Omni-30B, reaches 34.0% (English) and 19.3% (Mandarin), and the weakest tested model passes 1.6% of English cases.
Why do voice agents fail in multi-speaker settings according to the paper?
Three separable causes: open-weight models mostly cannot recover who said what from overlapping audio (clean transcripts lift them 21–44 points); frontier models mis-handle speaker-scoped decisions even from clean transcripts; and models across the board frequently answer questions no one addressed to them.
Where can I get MSI-Bench?
Per the paper's footnote, the dataset is on Hugging Face (M2cha4l1124/MSI-Bench) and the evaluation harness on GitHub (boson-ai/MSI-Bench), with serving configurations (vLLM builds, Docker images, 8×A100 hardware) documented in Appendix D.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–6 are accessible only via captions and in-text numbers; per-configuration transcript-lift values and full per-level SNR curves could not be independently read from the extracted text.
- Generation and judging prompts are not included in the text; pipeline claims (leakage gating, gold uniqueness) rest on the authors' description.
- Each configuration–case pair is judged once; run-to-run variance for hosted models is not reported in the extracted text.
- The reference list and Appendices A–D appear complete in the extraction; no evident truncation of the main text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.