ECHO: A Matched-Contrast Benchmark for Context-Sensitive Turn-Taking in Full-Duplex Dialogue
Shuofeng Zhao, Hongwei Cai, Wenke Fan, Qingxiang Guo, Dawei Yang, Zhou Wang, Zhiyang Zhou, Yingxin Shang, et al.
cs.CL
Quick verdict
ECHO proposes a matched-insertion paired benchmark exposing YIELD bias in full-duplex turn-taking, but its central 'context-sensitivity' claim is weakened by label-dependent interruption acoustic processing and the absence of a gain-free or waveform-reuse control.
Summary
ECHO is a paired Chinese benchmark for full-duplex turn-taking that holds the overlapping user utterance lexically fixed while rewriting the preceding multi-turn dialogue so the same insertion requires YIELD (interruption) or KEEP (backchannel, off-talk). Built from Duplex-Drama skeletons via DeepSeek-V4-Pro selection and Claude-3.5-Sonnet context rewrites, with human review at 97.45% accuracy, it contains 266 groups, 549 TTS-synthesized instances (183 per role), and 300 pair relations. A pairwise success rate requiring joint correctness on both pair members assigns no credit to constant-action policies. Evaluating four speech systems plus a text-only Gemini reference, three systems show severe YIELD bias (e.g., Easy Turn: 98.91% interruption yield but 1.09% backchannel keep; PASR 0.00%), MiniCPM-o 4.5 is balanced (54% PASR), and Gemini reaches 86.34% backchannel keep despite less assistant-side context than failing systems. Main limitations: interruption audio receives label-dependent RMS scaling and onset emphasis, and the paper concedes a waveform-reuse or gain-free ablation would be needed for strict acoustic control; systems also differ in context access and modality, preventing modality-matched comparison.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing turn-taking benchmarks contain a lexical shortcut (SID-Bench 81.8% top-15-character non-interruptive instances; disjoint Easy Turn vocabularies) that lets systems separate classes without using dialogue context, and ECHO's matched-insertion design removes it by construction.
- Most evaluated full-duplex speech systems exhibit a pronounced action-level YIELD bias: Easy Turn, SoulX-Duplug, and Lychee-FD keep the floor on only 1.09%, 6.56%, and 12.02% of backchannels despite high interruption yield rates, and their PASR is 0.00-4.00%.
- Interruption-only or sample-level evaluation overstates turn-taking reliability: Gemini's 82.88% overall accuracy drops to 52.00% PASR and its 82.33% macro accuracy to 66.00% PRSR under joint pair correctness.
- The bias is not explained by limited context access: the text reference that observes strictly less assistant-side information than Lychee-FD reaches 86.34% backchannel keep, implying the limiting factor is how floor decisions use context rather than how much context is available.
- Only MiniCPM-o 4.5 is comparatively balanced among speech systems (63.39/65.57/53.55 per-role accuracies, 54.00% PASR).
Strengths
Matched-insertion contrastive design removes the lexical shortcut in prior benchmarks
Sec. 2.4/Table 2 document that SID-Bench non-interruptive instances are 81.8% top-15-character only and Easy Turn has disjoint backchannel/turn-taking vocabularies, whereas ECHO's 100 interruption-backchannel pairs carry identical insertion text, 400-character backchannel vocabulary, and matched length distributions (6.3/6.2/6.6 chars).
Pair-level metrics are a genuine methodological contribution
PASR (Eq. 1) requires joint correctness on both members of a context-contrast pair, making constant-YIELD and constant-KEEP degenerate policies score zero; the complementary PKC guards against constant-KEEP, and the three-way PRSR extension is also defined.
Bias findings are concrete and quantified across systems
The text-reference comparison supports the 'usage not quantity of context' argument
Unusual transparency about construction confounds and AI use
Weaknesses
Label-dependent acoustic processing of interruptions is a potential cue leakage confound
No inferential statistics anywhere
Cross-system comparison is not modality- or interface-matched
LLM-in-the-loop construction with limited reported validation detail
Off-talk evidence is weak even by the authors' own reading
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ECHO benchmark?
ECHO (Evaluating Context-conditioned Handling of Overlaps) is a paired Chinese benchmark for full-duplex spoken dialogue systems. It contains multi-turn dialogues in which the same overlapping user utterance is inserted into two or three rewritten dialogue contexts that change its role — an interruption requiring the system to yield the floor, a backchannel or off-talk remark where the system should keep talking — plus pair metadata linking the variants. It includes 549 synthesized audio instances in 266 groups and 300 pair contrasts.
How is ECHO different from existing turn-taking benchmarks like Full-Duplex-Bench?
Prior benchmarks evaluate overlap events independently, and their non-interruptive feedback is often separable from interruptions by vocabulary alone (the paper reports 81.8% of SID-Bench non-interruptive instances use only the fifteen most frequent Chinese characters). ECHO instead holds the overlapping utterance text identical across a pair and rewrites only the preceding context, and it scores pair accuracy — credit only if the system acts correctly on both members — so a system with a fixed yield or keep habit scores near zero.
What are the main findings of the ECHO paper?
Three of four evaluated speech systems show a strong yield bias: Easy Turn, SoulX-Duplug, and Lychee-FD yield to 55-99% of interruptions but keep speaking through only 1-12% of backchannels, giving pairwise success rates of 0-4%. Only MiniCPM-o 4.5 is relatively balanced. A text-only Gemini reference reaches 86.34% backchannel keep with less assistant context than the failing systems, suggesting the problem is how context is used, not how much is available.
Is the ECHO dataset publicly available?
The paper states ECHO and its metadata will be publicly released, including the audio, the model-observable dialogue text, the original scenario labels, binary action labels, event timestamps, and the group/pair linking metadata. The generation prompts and evaluation code are not explicitly promised, so check the authors' release page for the final artifact set.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Fig. 1 and rendered audio examples cannot be inspected from the extracted text, so the qualitative contrast quality and acoustic rendering are judged from descriptions only.
- The evaluation harness, exact prompts, and any held-out splits are not described in the provided text, so reproducibility assessment relies on the stated release promise.
- The paper text appears complete (abstract through references), but page-level layout, appendix material, and per-example metadata could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.