RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialogue
Yuqi Wang, Fengyuan Liu, Haochen Luo, Zhiqi Yu, Qi Liu, Qi Liu
cs.CL · cs.AI
Quick verdict
Benchmark of 310 roles shows spoken role-playing breaks by ~turn 10; valuable resource, but single LLM judge and no error bars temper the headline claims.
Summary
The authors introduce RoleBreak, an open benchmark for evaluating long-horizon role-playing robustness in speech-to-speech dialogue systems. Five annotators curated 310 roles (character-based and user-centered) from five prior text role-playing benchmarks plus original personas, and designed multi-turn scenarios averaging 21.57 turns with context-dependent probes and stress interventions targeting identity, knowledge, agency, affect, interaction, safety, and protected information. LLMs drafted user turns and rubric criteria, humans revised, and TTS (CosyVoice voice cloning from LibriTTS-R/VCTK) rendered 6,688 user turns, of which 1,856 carry expressive emotion targets; 11,743 turn-level criteria were human-verified (mean quality 4.42/5). Nine configurations were evaluated: PersonaPlex (full-duplex), four omni-modal models (Qwen2.5-Omni-7B, Qwen3-Omni-30B, MiniCPM-o-4.5, Covo-Audio-Chat), and cascaded ASR–LLM–TTS with Qwen3.5 scaled 2B→27B. DeepSeek-V4-Pro judges criterion pass rates; judge–human agreement was 94.55% on 312 samples. Key results: the strongest system (Qwen3.5-27B cascade) hits first persona failure at turn 10.4 and safety failure at 11.6, versus 21.6 average conversation length; all systems score below 14.7/100 on vocal emotion; LLM scaling improves semantic metrics but not emotion; and user speech emotion shifts downstream persona/interaction/safety scores by 4.2–5.6 points with fixed content. Limitations include single-judge scoring without confidence intervals, synthetic user speech, and exclusion of commercial closed systems.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 19 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Current speech-to-speech systems are far stronger at semantic role adherence than at vocal emotion: all nine evaluated configurations score below 14.7/100 on vocal emotion (Table 2).
- Semantic role-playing robustness is brittle over long horizons: even the strongest system (Qwen3.5-27B cascade) encounters its first persona failure at turn 10.4 and safety failure at 11.6 on average, versus 21.57 average turns per dialogue.
- Scaling the cascaded LLM from 2B to 27B substantially improves semantic robustness (persona 58.6→80.3, safety 42.2→71.7) but yields little vocal-emotion improvement (13.0→14.1).
- User vocal emotion alone shifts downstream role-playing behavior by up to 5.6 points (persona, Table 3) even when linguistic content is held fixed.
Strengths
Unusually thorough human verification
Five annotators curate roles and scenarios; five reviewers rate retained examples on four quality axes, yielding 4.42/5 mean and 90.36% rated ≥4 (§3.2) — a verification depth most spoken-dialogue benchmarks (Table 1 comparisons) lack.
First-failure-turn metrics expose what aggregate pass rates hide
P-FFT/S-FFT (§4.2) reveal that the best aggregate system (Qwen3.5-27B: persona 80.3, safety 71.7) still fails at turn 10.4/11.6 of a 21.57-turn average dialogue — a genuinely informative long-horizon lens absent from prior speech benchmarks in Table 1.
Clean controlled scaling experiment
Well-motivated affect intervention
Public artifacts and validated judge
Weaknesses
FFT metrics are highly sensitive to un-decomposed judge noise
No statistical inference anywhere
'Current systems' claims rest on a narrow pool
Vocal-emotion conclusion is partly confounded in cascades
All user speech is synthetic; minor citation sloppiness
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
RoleBreak (arXiv 2609.16614, University of Hong Kong, 4-page IEEE-format paper) is an open benchmark for **long-horizon role-playing robustness in speech-to-speech dialogue**. The paper advances four claims: (1) semantic role adherence vastly exceeds vocal-emotion capability — all nine evaluated systems score below 14.7/100 on emotion (Table 2); (2) semantic robustness is brittle over long interactions — the strongest system (Qwen3.5-27B cascade) hits its first persona failure at turn 10.4 and safety failure at 11.6, against a 21.57-turn average dialogue; (3) LLM scaling improves semantics (persona 58.6→80.3 from 2B→27B) but not vocal emotion (13.0→14.1); (4) user vocal emotion alone shifts role-playing behavior (Table 3). Claims (1)–(3) are well supported *within the evaluated pool*; claim (4) is directionally convincing but statistically under-instrumented; all four are generalized to "current systems" beyond what nine mostly-open configurations license.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the RoleBreak benchmark?
RoleBreak is an open benchmark from University of Hong Kong researchers for testing whether speech-to-speech dialogue models can hold a persona over long, adversarial conversations. It contains 310 roles (both fictional characters and user-centered personas), 6,688 human-verified dialogue turns averaging 21.57 turns per conversation, 11,743 fine-grained evaluation criteria, and 1,856 turns with expressive emotion targets. It is released on Hugging Face and GitHub.
Which spoken dialogue model performed best on RoleBreak?
The cascaded ASR–LLM–TTS system using Qwen3.5-27B scored best on semantic metrics (persona 80.3/100, safety 71.7/100, interaction 62.6/100), but even it met its first persona-criterion failure around turn 10.4 and first safety failure around turn 11.6 of a 21.57-turn average conversation. No system scored above 14.7/100 on vocal emotion.
How reliable is the AI judge used to score RoleBreak?
The authors used DeepSeek-V4-Pro to decide whether each of the 11,743 criteria is satisfied. On 312 randomly sampled rubric–response pairs, each judged independently by three human annotators, the judge agreed with humans 94.55% of the time. The paper does not report agreement per evaluation dimension or the direction of disagreements, which matters because the first-failure-turn metric triggers on any single criterion failure.
What are the four main findings of the RoleBreak paper?
(1) All nine evaluated systems are much better at semantic role adherence than at vocal emotion (emotion scores all below 14.7/100). (2) Long-horizon robustness is weak: the best system's first persona and safety failures occur at turns 10.4 and 11.6 on average. (3) Scaling the LLM from 2B to 27B in a fixed cascaded pipeline greatly improves semantic metrics but barely moves vocal emotion. (4) Changing only the emotional tone of the user's voice — with identical words — shifts the model's role-playing scores by 4.2 to 5.6 points.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review is based on the extracted arXiv text; Figures 1–2 (construction pipeline and example turns) could not be analyzed, and the paper has no visible appendix.
- Several evaluated components (PersonaPlex, Covo-Audio-Chat, DeepSeek-V4-Pro, Qwen3.5, Qwen3-TTS) are cited as 2025–2026 preprints whose properties could not be independently verified from this text.
- Dataset and code URLs are stated in the paper but were not inspected; completeness of the released artifacts is unverified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.