Enabling Streaming User Transcription in Full-Duplex Speech-to-Speech Models
Ke Hu, Nourchene Ferchichi, Edresson Casanova, Ankita Pasad, Elena Rastorgueva, Chen Chen, Nithin Rao Koluguri, Piotr Zelasko, et al.
cs.CL
Quick verdict
A parallel ASR head gives a duplex S2S model streaming transcription at 10.21% WER with turn-taking intact; sensible method, thin statistics.
Summary
The paper adds streaming user speech transcription to a full-duplex speech-to-speech (S2S) model built on SALM-Duplex: a 600M Parakeet streaming encoder feeds a 9B Nemotron-Nano-9B-v2 backbone, and a new ASR head — a separate embedding layer and prediction head initialized from the backbone — predicts user transcription in parallel with the agent text head, in a single decoding pass. Training uses on-the-fly CTC forced alignment (MMS-FA) to place word-start-aligned tokens on 80ms frames, with a 1.2s user text delay and 0.16s agent text delay; data comprises 16k hours of English ASR data plus interleaved conversation data with noise augmentation. The duplex model reaches 10.21% average WER on the HuggingFace Open ASR Leaderboard (Table I) — better than the FastConformer streaming baselines cited — while turn-taking precision (90% vs 86.1%), barge-in accuracy (100%), and VoiceBench scores remain close to the no-ASR baseline. The same architecture trained standalone achieves 8.47% average WER at 1.6s latency, improving to 7.73% with Granary-derived YODAS/YTC data (Table V), still behind Nemotron-Speech-0.6B (7.16%) and Qwen3-ASR (5.76%). Main limitations: turn-taking metrics come from an internal ~60-conversation test set with no significance testing; several training data sources are in-house or partially undisclosed; code is promised but not yet released; and the parameter/compute overhead of the added head is never quantified.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A lightweight ASR head in parallel to the agent text head adds streaming user transcription to an existing duplex S2S model with no significant architectural change to the base model.
- The duplex S2S model with the ASR head achieves 10.21% average streaming WER on the HuggingFace Open ASR Leaderboard while preserving turn-taking and barge-in performance.
- The same architecture trained as a standalone streaming ASR model achieves 7.73% average WER, competitive with SOTA streaming ASR at lower latency than Qwen3-ASR or Kyutai STT.
- Adding the ASR head does not significantly change turn-taking behavior, and slightly improves OpenbookQA accuracy (66.59% to 69.01%), suggesting a benefit from the text modality.
Strengths
Clean, minimal architectural extension
Section III-B adds only a separate embedding layer and prediction head, both initialized from the Nemotron backbone, so a single decoding pass jointly emits user transcription and agent text — no changes to the 9B decoder or the 80ms streaming encoder path.
Full-duplex behavior is preserved and partly improved
Table II: turn-taking precision rises 86.1% to 90% and barge-in latency improves 393ms to 374ms vs the no-ASR baseline; Table IV shows far fewer false takeovers during pauses than Moshi (TOR 44.4% vs 98%).
Unusually candid trade-off reporting
Standalone model is independently useful
Reproducibility intent
Weaknesses
Turn-taking evidence rests on a closed internal test set
Selective baseline framing in the duplex headline
Cost of the head never quantified
Unexplained latency ablation anomaly
Partially closed data and absent code
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Peer Review: Enabling Streaming User Transcription in Full-Duplex Speech-to-Speech Models
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What WER does this full-duplex model achieve on speech recognition?
When the streaming ASR head is integrated into the full-duplex S2S model, it reaches 10.21% average WER on the HuggingFace Open ASR Leaderboard test sets. The same architecture trained as a dedicated standalone streaming ASR model achieves 8.47% average WER at 1.6s latency, improved to 7.73% by adding YouTube-derived (YODAS/YTC) training data.
Does adding the transcription head hurt the conversation quality of the duplex model?
Mostly no, per the paper's internal evaluation: turn-taking precision improved from 86.1% to 90%, barge-in stayed at 100% accuracy, and VoiceBench OpenbookQA rose from 66.59% to 69.01%. Small regressions were CommonEval (3.24 to 3.11 out of 5) and the smooth turn-taking latency on FDB-v1, which rose from 257ms to 477ms.
Is the code or the model available?
Not yet. The paper states the authors will open-source training and inference code, but no repository link is given, and part of the training data (in-house ASR data and a subset of Granary) is not publicly described. Reproducibility therefore currently depends on the promised release.
How does this compare to Moshi, which also does full-duplex conversation?
Moshi provides no user transcription at all — that is the gap this paper fills. On the FDB-v1 benchmark, this model shows much better response quality when interrupted (GPT score 3.99 vs 0.77) and far fewer false takeovers during user pauses (44.4% vs 98%), but its smooth turn-taking is slower (477ms vs 265ms latency).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 appears in the extracted text only as garbled inline labels; architectural reading is based on the caption and body text rather than the rendered figure.
- The provided metadata lists 8 authors while the paper text lists 12 (adding Yifan Peng, Hainan Xu, Zhehuai Chen, Boris Ginsburg); this review follows the full text.
- Code is promised ('we will open-source') but no repository exists yet, so reproducibility claims could not be verified against an artifact.
- The evaluation protocol for Table I (whether agent audio/text was active during leaderboard scoring) is not described in the text and could not be inferred.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.