Neither Silence nor Overlap Is Failure: Intent-Conditioned Evaluation of Turn-Taking in Full-Duplex Spoken Dialogue Models
Kian Shamsaie, Iman Modarressi
cs.CL · cs.AI · cs.HC · cs.SD
Quick verdict
Proposes TACT: intent-conditioned proper-scoring turn-taking benchmark with solid theory and striking model gaps, though LLM-judge and VAP circularity remain.
Summary
The paper argues that binary fixed-window metrics misjudge turn-taking in full-duplex spoken dialogue models because a response offset's appropriateness depends on the speaker's latent intent. It introduces TACT, a benchmark of 9,728 episodes (73.2 hours) from five public dyadic corpora plus synthetic probes, where each episode carries dialogue history, a per-speaker memory profile, and a six-class intent posterior fusing 3-5 human annotations with a calibrated LLM judge. Scoring replaces binary windows with a threshold-weighted continuous ranked probability score (twCRPS) whose weights derive from intent-conditioned timing kernels fitted to human floor-transfer-offset distributions; the authors prove boundedness, strict propriety, reduction of Full-Duplex-Bench to a degenerate limit, and mixture identifiability. Across eleven systems the best (gpt-realtime-2) reaches 0.47 against a human topline of 0.86; rankings reorder versus reproduced binary metrics (Spearman 0.55); all systems show a two-sided over-eagerness pathology (withholding after rhetorical turns in under 24% of episodes versus 74% for humans); and a memory-swap ablation finds no per-speaker adaptation (JSD squared at most 0.024 versus 0.23 for humans). The composite correlates with held-out human judgments at Spearman 0.81 versus 0.46 for binary metrics. Limitations include label noise (alpha 0.73), English-only kernels, dependence on a VAP-style completion estimator for overlap scoring, possible judge circularity, and a qualitative, non-quantified reconciliation of the 0.86 topline with the theoretical optimum of 1.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The appropriateness of a response offset (silence, delayed response, or anticipatory overlap) in dyadic dialogue is conditional on the focal speaker's latent intent, which is identifiable from that speaker's behavior across episodes, so evaluation must condition on intent posteriors and per-speaker memory profiles.
- A threshold-weighted CRPS with intent-conditioned timing kernels is strictly proper with the human conditional law as its unique maximizer, and binary fixed-window metrics such as Full-Duplex-Bench arise as a degenerate weight limit of it.
- Current full-duplex SDMs exhibit a two-sided over-eagerness pathology — near-uniform fast response regardless of intent plus inability to launch invited cooperative overlap — with the best system reaching 0.47 against a human topline of 0.86.
- Model timing behavior is essentially invariant to speaker memory profiles (composite change at most 0.01, JSD squared at most 0.024) whereas held-out human behavior is not (JSD squared 0.23), demonstrating an absence of per-speaker adaptation.
- The TACT composite agrees with held-out human judgments at Spearman 0.81 versus 0.46 for the binary metrics it replaces, and 0.62 for an intent-conditioned binary baseline.
Strengths
Real scoring theory, not metric relabeling
Proposition 1 (Sec. IV-E) gives a complete bias-variance + Tonelli proof that twCRPS is strictly proper with the human conditional law as unique maximizer, correctly handling the no-response atom via the indicator convention; Proposition 2 shows binary fixed-window scoring is a degenerate delta-limit of the weight function, subsuming Full-Duplex-Bench rather than strawmanning it.
Multi-pronged validity evidence with component ablations
Sec. VI-D: over 88 system-condition cells on 1,280 held-out episodes, TACT reaches Spearman 0.81 (95% CI [0.77, 0.84]) vs 0.46 for reproduced binary metrics, 0.39 for raw latency, and — crucially — 0.62 for an intent-conditioned binary baseline, isolating the contribution of continuous scoring; component ablations (memory 0.71, calibration 0.78, prosody 0.74) attribute the gains.
Concrete, falsifiable diagnostic findings
Construct validity signals in the aggregate numbers
Ecologically diverse, stratified data design
Weaknesses
Proposition 3 contradicts the paper's own thesis
Topline reconciliation asserted, never quantified
Two circularity channels in scoring
Statistical reporting gaps
Reproducibility weak links
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
TACT addresses turn-taking evaluation for full-duplex spoken dialogue models (SDMs). The central construct: the appropriateness of a response offset — silence, fast response, or anticipatory overlap — is conditional on the focal speaker's latent intent, identifiable from that speaker's behavior across episodes. The paper makes three formal claims (boundedness and strict propriety of the threshold-weighted CRPS in Proposition 1; reduction of binary fixed-window scoring to a degenerate weight limit in Proposition 2; intent-mixture identifiability in Proposition 3) plus an empirical set: current SDMs are uniformly over-eager, lack per-speaker timing adaptation, and the intent-conditioned composite tracks human judgment far better (Spearman 0.81) than the binary metrics it replaces (0.46). The positioning against Full-Duplex-Bench v1-v3, FD-Bench, and Talking Turns (Sec. II) is accurate, and the contribution claim is appropriately narrowed to intent-conditioning plus proper scoring rather than multi-turn or overlap awareness per se.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the TACT benchmark?
TACT (Theory-of-mind Aware Conversational Timing) is a benchmark for evaluating turn-taking in full-duplex spoken dialogue models. It contains 9,728 episodes totaling 73.2 hours, drawn from five public dyadic conversation corpora (CANDOR, SSSD, otoSpeech, Seamless Interaction, AMI) plus 768 synthetic probes. Unlike binary fixed-window benchmarks, each episode is scored against a posterior over six speaker intents (immediate answer, reflective, rhetorical, floor-holding, backchannel-inviting, abandonment), using a threshold-weighted continuous ranked probability score whose weights come from human timing distributions, together with the speaker's dialogue history and a memory profile of that speaker.
Why do current full-duplex speech models score poorly on TACT?
Because they respond almost uniformly fast regardless of the speaker's intent. Humans wait when a turn is rhetorical or floor-holding (withholding a response 74% of the time after rhetorical turns), while the best models withhold less than 24% of the time; humans also launch 63% of invited backchannels in overlap, whereas models rarely do (14% at best). The best system, gpt-realtime-2, reaches 0.47 against a human topline of 0.86, and a memory-swap test shows models do not adapt their timing to individual speakers while humans do.
How much better is TACT than binary metrics at matching human judgment?
In a validity study over 88 system-condition cells using 1,280 held-out episodes, the TACT composite correlates with held-out human judgments at Spearman rho = 0.81 (95% CI [0.77, 0.84]), versus 0.46 for reproduced binary Full-Duplex-Bench-style metrics, 0.39 for raw response latency, and 0.62 for an intent-conditioned binary baseline — showing that continuous scoring, not intent conditioning alone, drives most of the gain.
Which models were evaluated on TACT and how did they rank?
Eleven systems: dGSLM, Moshi, SyncLLM, Freeze-Omni, Qwen2.5-Omni, MiniCPM-o 2.6, PersonaPlex, GPT-4o Realtime, gpt-realtime-2, and Gemini 3.1 Flash Live at minimal and high thinking, plus VAP-gated cascade, never-respond, and human-topline baselines. gpt-realtime-2 ranked first (0.47); notably, Moshi fell from second place under binary metrics (thanks to its 0.26 s latency) to tenth under TACT, because instant takeover fails on reflective, rhetorical, and floor-holding turns.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as embedded text; Fig. 1 kernel shapes and the Fig. 2 heat-map were assessed from the numeric cell values reproduced in the text.
- The Supplementary Material (data, kernels, prompts) is referenced but not included in the extracted text, so availability and annotation guidelines could not be verified.
- Mathematical notation in Eqs. (1)-(6) and the Proposition 1-3 proofs may contain minor text-extraction artifacts; the proofs were assessed from the extracted prose.
- Proprietary system versions (gpt-realtime-2, Gemini 3.1 Flash Live) and the LLM judge identity cannot be independently confirmed from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.