A frontend-backend architecture for tool calls in full-duplex speech models
Ke Hu, Slyne Deng, Chen Chen, Elena Rastorgueva, Edresson Casanova, Punit Kumar, Dharmendra Choudhary, Nikhil Srihari, et al.
cs.CL
Quick verdict
Token-based delegation adds tool calls to a duplex speech frontend; strong benchmark results, but no significance testing and low reproducibility.
Summary
The paper proposes a frontend-backend architecture that adds tool calling to full-duplex speech-to-speech systems while keeping the speech model largely unchanged. A duplex speech-to-text frontend (a Parakeet streaming encoder feeding a Nemotron-Nano-9B backbone) is trained to emit a delegation token when an utterance requires a tool, say a short filler phrase, and forward streaming ASR transcripts to a LangGraph ReAct backend running Qwen2.5-7B up to Qwen3-235B-A22B; the backend's answer is prefilled into the frontend's agent-text channel, repeated verbatim, and spoken through streaming TTS. Training uses roughly 530k hours of pretraining, 111k hours of SFT, and 8.5k hours of synthetic multi-turn tool-call audio. On audio BFCL the system reaches 74.6% average AST accuracy with 81.2% irrelevance rejection and 92-97% delegation-token recall. On Full-Duplex-Bench-v3 the 235B backend is comparable to GPT-realtime-mini; on EVA-Bench it outperforms GPT-realtime-mini and Qwen3-Omni-30B but trails GPT-realtime-2, with median turn-taking latency of 3.14 s versus 1.65 s. Side-effect tables show small regressions in ASR WER (10.80 to 11.47%), CommonEval, and pause handling. Limitations include absent statistical significance testing, author-reproduced baselines under non-uniform settings, a judge model (GPT-5.2) that doubles as the topline baseline, massive non-released training data, and no code beyond an online demo.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A duplex STT frontend can learn to delegate tool calls via a single control token (<tc_bos>) with minimal architectural change, achieving 92-97% recall on tool-call utterances.
- The full system rejects 81.2% of irrelevant requests and reaches 74.6% average BFCL AST accuracy with a 30B backend plus external ASR.
- With a Qwen3-235B-A22B backend, the system is competitive with GPT-realtime-mini on FDB3 and outperforms GPT-realtime-mini and Qwen3-Omni-30B-A3B on EVA-A/EVA-X.
- Tool-call training largely preserves duplex behavior, with only small regressions in ASR WER, CommonEval, turn-taking precision/recall, and pause handling.
Strengths
Precisely specified delegation protocol
Section 4 gives exact token semantics and timings (<tc_bos> replaces <agent_bos> with ~320 ms extra latency, ~1 s filler, <tc_eos>, prefill ~1 s later, loss-masked) plus a full example training sequence — enough detail for another lab to replicate the recipe.
Honest side-effect reporting
Tables 4-7 quantify what tool-call training costs: ASR WER 10.80→11.47%, CommonEval 2.87→2.36, turn-taking precision/recall 85/94→82/91, FDB-v1 pause handling worse (53.2→68.2), alongside gains in smooth TT (100 vs 93) and barge-in latency (349 vs 590 ms).
Strong delegation and rejection numbers
Useful ablations
Broad, multi-benchmark evaluation
Weaknesses
No statistical inference anywhere
Judge and baseline circularity in EVA
Non-uniform baseline configurations
High interruption and filler rates unvalidated
Reproducibility and latency gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This NVIDIA systems paper targets a real gap: duplex speech-to-speech (S2S) models handle natural turn-taking but are weak at tool use, while text LLMs are strong agents. The proposal is a frontend-backend split (Sect. 3): a duplex speech-to-text frontend (600M Parakeet streaming encoder + Nemotron-Nano-9B-v2-Base, with parallel streaming-ASR and agent-text heads) is fine-tuned to fire a delegation token `<tc_bos>` when the user's utterance plausibly needs a tool, emit a ~1 s filler, and forward endpointed streaming ASR transcripts to a LangGraph ReAct backend (Qwen2.5-7B up to Qwen3-235B-A22B). Backend answers are prefilled into the agent-text channel between `<pf_bos>`/`<pf_eos>`, repeated verbatim, then voiced by VoiceChat-TTS. The central claims (Sect. 1, 6) are that delegation adds tool calls with 92-97% token recall and 81.2% irrelevance rejection, matches GPT-realtime-mini on FDB3, beats it and Qwen3-Omni-30B on EVA-Bench, and largely preserves duplex behavior. On the reported numbers the claims are directionally supported; the evidentiary layer, however, is thin on statistics.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How do full-duplex speech models make tool calls in this paper?
The duplex speech-to-text frontend is trained to emit a special delegation token (<tc_bos>) when the user's utterance needs a tool, say a one-second filler phrase, and then hand the streaming ASR transcript to a text LLM backend (a LangGraph ReAct agent). The backend's answer is injected back into the frontend's text stream via a prefill-and-repeat mechanism and spoken aloud by streaming TTS.
What are the headline results of this NVIDIA frontend-backend speech paper?
On the speech version of BFCL, the system achieves 74.6% average AST accuracy (30B backend), rejects 81.2% of irrelevant requests, and fires the delegation token correctly on 92-97% of tool-call utterances. On EVA-Bench, the 235B-backend system scores 46.6 EVA-A versus GPT-realtime-mini's 33.1, but trails GPT-realtime-2 (59.4) and the text-only GPT-5.2 topline (63.3).
Is code or a model available for this paper?
The paper links only an online demo (a Hugging Face Space). No release of code, model weights, or the roughly 700k hours of training data is stated, so independent full reproduction is not currently possible; the evaluation benchmarks themselves (audio BFCL, Full-Duplex-Bench v3, EVA-Bench) are public.
Does adding tool-call training hurt the model's normal conversation ability?
Slightly, according to the paper's side-effect tables: ASR word error rate rose from 10.80% to 11.47%, CommonEval fell from 2.87 to 2.36 (of 5), turn-taking precision/recall dropped from 85/94 to 82/91, and pause handling degraded on FDB-v1, while smooth turn-taking and barge-in latency actually improved.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Fig. 1 is available only as OCR fragments; the architecture was reconstructed from prose and the caption.
- The metadata author list is truncated: the full text lists 13 authors plus acknowledgments, metadata only 8.
- Citations are future-dated relative to review time (2026 venues and models); external claims such as tau-Voice's 31-51% figure could not be independently verified.
- Tables 4-7 rely on an internal benchmark that is neither described in detail nor released; those numbers are unverifiable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.