DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents
cs.AI
Quick verdict
Rigorous benchmark for implicit instruction following in duplex voice agents; sound design, but n=30 conflicts and no significance tests limit conclusions.
Summary
The paper introduces DuplexSpeechBench-IFEval (DSB-IFEval), a benchmark of 1,038 test cases (240 unique two-turn user-side conversations, TTS-synthesized with injected pauses, chunk gaps, and barge-ins) spanning eight assistant roles and five conditioning protocols: bare default (L0), explicit behavioral instruction (L1), persona-only (L2), persona plus restated rule (L3), and benign (L4a) or safety (L4b) instruction conflict. Floor management is scored deterministically via an Instruction Adherence Score (IAS) from known probe timestamps and Silero VAD, while persona-consistent content is judged by gpt-4o as a Persona Adherence Score (PAS); ENTAILMENT GAP (L1−L2), Redundancy Gain, and Role Tax summarize conditioning effects. Across six systems (GPT-Realtime, MiniCPM-o-4.5, Fun-Audio-Chat-8B, PersonaPlex-7B, F-Actor, and persona-blind Moshi), full-duplex models show the largest explicit-to-persona gaps (F-Actor +9.7pp, PersonaPlex +4.5pp), whereas GPT-Realtime and the turn-based systems show little IAS change but much stronger persona-consistent content; Role Tax is non-positive for all. Under safety conflict, only MiniCPM-o reaches majority safety-override (60.0% of 30 cases) and four systems score 0.0% IAS at L4b. Limitations: English-only, two-turn, synthetic TTS user speech, a single unvalidated LLM judge, a 30-case conflict subset, no statistical tests, and aggregate IAS conflating architecture with competence.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- DSB-IFEval is the first benchmark to jointly evaluate explicit turn-management instruction following, persona-implied floor behavior, and instruction hierarchy in real-time full-duplex spoken interaction, via 1,038 cases across eight roles and five conditioning protocols.
- Full-duplex models are most sensitive to whether behavior is stated or inferred: F-Actor loses 9.7pp and PersonaPlex 4.5pp of adherence under persona-only conditioning, while GPT-Realtime, MiniCPM-o, and Fun-Audio-Chat change little.
- Persona-consistent content (PAS) and real-time floor control (IAS) are distinct capabilities, so aggregate instruction-following scores conflate what an agent says with how it manages the floor.
- Adding a persona never improves execution of an already-explicit instruction (Role Tax ≤ 0 for all six systems) and restating the implied rule yields at most +2.0pp.
- Systems that resolve benign conflicts correctly (up to 89.9% directive-wins) fail safety conflicts: only MiniCPM-o-4.5 overrides the directive for safety in a majority of cases (60.0%).
Strengths
Matched-audio controlled design isolates conditioning effects
Each synthesized conversation is reused across L0–L4b with identical waveform, speaker, and probe timing (§3.2), and a byte-identical matched-audio triad (R1/R2/R3, §C.5, Table 6) holds the user signal fixed across roles, so only the conditioning prompt varies.
Deterministic floor scoring separated from LLM-judged content
IAS verifiers operate on construction-time probe timestamps plus Silero VAD (§4, §E), not LLM transcript reading; raw timings are stored so thresholds are recomputable, exercised in a 180-configuration sweep (Table 19: mean Kendall's τ 0.796, top-1 preserved 180/180).
Persona-blind negative control with audited leakage gates
Genuinely diagnostic dissociation findings
Unusually complete reporting and release plan
Weaknesses
No inferential statistics anywhere
Architecture confound in aggregate IAS
Single unvalidated LLM judge for PAS and conflict labels
Moshi control behaves anomalously
Limited ecological validity and thin conflict protocol
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper introduces **DuplexSpeechBench-IFEval (DSB-IFEval)**, a benchmark for *implicit* instruction following in full-duplex voice agents: whether a system can infer floor-management behavior (listen, backchannel, interrupt, yield, continue through overlap) from a role/persona rather than an explicit rule, execute it at the right conversational moment, and prioritize correctly when instructions conflict. It contains 1,038 test cases from 240 unique two-turn user-side conversations spanning eight behaviorally contrastive roles and five conditioning protocols (L0 default, L1 explicit instruction, L2 persona-only, L3 persona plus restated rule, L4a benign / L4b safety conflict). Positioning against INSTRUCT-FD, the Full-Duplex-Bench line, and persona evaluations is tabulated explicitly (Table 1). The three headline claims — entailment gaps concentrated in full-duplex models (F-Actor +9.7pp, PersonaPlex +4.5pp), dissociation of persona content (PAS) from floor control (IAS), and fragile safety hierarchies — are clearly stated and descriptively supported, though never tested inferentially.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DuplexSpeechBench-IFEval?
A benchmark for testing whether full-duplex voice agents follow instructions implicitly. It contains 1,038 test cases built from 240 synthetic two-turn user conversations across eight assistant roles (e.g., grief counselor, 911 dispatcher, interpreter) and five conditioning protocols ranging from bare defaults to persona-only conditioning and persona-versus-directive conflicts. Floor behavior is scored deterministically (IAS) from known event timestamps and voice-activity detection; persona-consistent content is scored by an LLM judge (PAS).
Which speech models were evaluated and what were the main findings?
Six real-time systems: GPT-Realtime, MiniCPM-o-4.5, Fun-Audio-Chat-8B, PersonaPlex-7B, F-Actor, and Moshi (as a persona-blind control). Full-duplex models (F-Actor, PersonaPlex) lose the most adherence when behavior must be inferred from a persona (+9.7 and +4.5 percentage-point entailment gaps), while GPT-Realtime and turn-based systems keep stable floor behavior but produce more persona-consistent content. Under safety conflicts, only MiniCPM-o overrode a conflicting directive for safety in a majority of cases (60% of 30).
How does DSB-IFEval differ from INSTRUCT-FD?
INSTRUCT-FD evaluates explicit turn-management instructions using transcript-based judging. DSB-IFEval reuses its IAS concept but scores timing deterministically from construction metadata, and adds persona-only conditioning (L2), persona-plus-restated-rule (L3), and instruction-conflict protocols (L4a benign, L4b safety) that test whether role-implied behavior can be inferred and correctly prioritized against explicit directives.
Is the benchmark data and code available?
The paper lists a project page (dsb-ifeval.github.io) and a release checklist stating that personas, user scripts and WAVs, event manifests, the deterministic scorer with raw timing measurements, judge prompts and outputs, model adapters, and analysis scripts are included, which the authors state is sufficient to reconstruct the benchmark and regenerate all reported automatic metrics.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure graphics (Fig. 1-5) are flattened to captions and raw number runs in the extraction; Figure 5 values cross-check against Tables 25-26, so no data loss was identified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.