NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
Jagadeesh Balam, Travis Bartley, Edresson Casanova, Edresson Casanova, Sanjay Chauhan, Chen Chen, Chen Chen, Zhehuai Chen, et al.
cs.CL · cs.AI
Quick verdict
First open full-duplex S2S model with native tool calling; strong turn-taking, but weak argument accuracy (42.2%, Pass@1 33.0%) and untested design claims.
Summary
This NVIDIA technical report presents NemotronLabs VoiceChat, an open full-duplex speech-to-speech model that adds native tool calling to continuous listening-speaking interaction. The system couples a 600M-parameter cache-aware FastConformer streaming encoder with a Nemotron-Nano-9B-v2 LLM backbone, parallel specialized output channels for agent text and structured function calls (<SOTC>/<EOTC>/<EOTR> protocol), an auxiliary RNN-T branch for incremental user transcription, and a separately trained 977M-parameter streaming TTS decoder (VoiceChat-TTS). Training proceeds in two stages (CPT then SFT) on 64 GPUs, mostly over TTS-rendered pseudo-dialogues with augmentation for interruptions, backchannels, and noise. On Full-Duplex-Bench 1.0 the model reports the lowest pause-handling takeover rates among open-weight systems (15.3% synthetic, 25.5% CANDOR), 100% interruption takeover with 4.33/5 response quality; on FDB 1.5 it resumes after user backchannels 93% of the time. VoiceBench averages 55.1, behind cascaded DuplexCascade (65.4) and MiniCPM-o 4.5 (76.1, different judge). On FDB 3.0, tool-selection F1 (82.5%) beats Gemini Live endpoints, but argument accuracy (42.2%) and Pass@1 (33.0%) trail substantially. Limitations include ~2-minute audio context, degraded tool use beyond roughly five tools, no barge-in during tool execution, and noisy-environment fragility. Notably, the central parallel-stream design claim is never compared head-to-head against DuplexSLA.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Full-duplex interaction, speech recognition and generation, general language capability, and external tool use can be integrated in a single open speech-to-speech model without sacrificing real-time conversational behavior (Abstract, Sec 6).
- Parallel specialized output streams for agent text and function calls preserve low-latency full-duplex behavior, unlike DuplexSLA's serialized action channel (Sec 1, 2.2).
- On FDB 1.0 the model achieves the lowest pause-handling takeover rates among open-weight systems (15.3% synthetic / 25.5% CANDOR) and the best post-interruption response quality in the comparison (4.33/5) (Table 1).
- On FDB 3.0 the model attains 82.5% tool-selection F1, above Gemini Live 2.5 (78.6) and 3.1 (81.7), while argument accuracy (42.2%) and Pass@1 (33.0%) remain below those baselines (Table 4).
- It is the first fully open full-duplex speech model to support tool calling (Sec 5).
Strengths
Genuinely unified architecture integrating four capabilities in one streaming stack
Sec 2 and Figure 1 combine streaming listening, incremental RNN-T transcription, reasoning, structured tool emission, and streaming TTS on a shared 80-ms timeline; the dedicated function channel with <SOTC>/<EOTC>/<EOTR> boundaries (Sec 2.2) and negative padding supervision (weight 0.3, Sec 3.2) is a concrete mechanism against spurious calls.
Strong, benchmark-verified turn-taking profile
Table 1: lowest open-weight pause-handling TOR (15.3% synthetic / 25.5% CANDOR), 100% interruption takeover with best-in-comparison 4.33/5 quality; Table 2: 93% Resume after backchannels, exactly matching Gemini Live 2.0 and 13 points above Freeze-Omni (80%).
Unusually honest reporting of weak results
Recipe-level reproducibility detail
Weaknesses
Central design claim never tested experimentally
Tool calling is selection-only strong
Benchmark-targeted training data
Nearly all training signal is TTS-synthesized
Editorial inconsistency undermines the 'completely open' claim
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This NVIDIA technical report (arXiv 2609.21967, September 2026) presents NemotronLabs VoiceChat, described as the first fully open full-duplex speech-to-speech model with native general tool calling. The scope is a complete system: a streaming speech encoder, a 9B decoder-only LM with parallel agent-text and function-call output channels, an auxiliary RNN-T transcription branch, and a separately trained streaming TTS decoder. Five claims are staked: (1) full-duplex interaction, speech recognition/generation, general language ability, and tool use integrate in one open model 'without sacrificing real-time conversational behavior' (Abstract); (2) parallel specialized streams preserve low latency better than DuplexSLA's serialized action channel (Sec 1); (3) best-in-class turn-taking among open-weight systems (Table 1); (4) tool-selection F1 above Gemini Live endpoints (Table 4); (5) priority as the first open full-duplex tool-calling model. A presentation defect runs through Sec 5: the system is repeatedly called 'V-Model', evidently residue of an anonymized draft.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is NemotronLabs VoiceChat?
It is an open full-duplex speech-to-speech model from NVIDIA that listens and speaks continuously (rather than waiting for silence) and can invoke external tools mid-conversation. It combines a streaming speech encoder, a 9B language model backbone, parallel output channels for agent text and structured function calls, an auxiliary branch that transcribes user speech, and a streaming text-to-speech decoder. The paper states the checkpoint is available on Hugging Face.
How well does it handle interruptions and turn-taking?
On Full-Duplex-Bench 1.0 it shows the lowest pause-takeover rates among open-weight systems (15.3% on synthetic pauses, 25.5% on CANDOR), takes the turn after 100% of user interruptions with a 4.33/5 post-interruption response quality, and on FDB 1.5 it resumes its original response after user backchannels ('uh-huh') in 93% of cases — exactly matching Gemini Live 2.0 on that condition. The closed GPT-Realtime endpoint still beats it on pause handling (12.0% takeover) and smooth turn-taking (100% vs 81.5%).
Can the model reliably call tools?
Only partially. Its tool-selection F1 on Full-Duplex-Bench 3.0 (82.5%) beats Gemini Live endpoints (78.6% and 81.7%), but argument accuracy (42.2% vs ~59%) and end-to-end success (Pass@1 33.0% vs 49-54%) trail them substantially. The authors recommend exposing no more than about five tools per session, note unreliable simultaneous multi-tool calls, and state that users cannot barge in while a tool is executing.
Is the model actually open and reproducible?
Partially. The paper claims the checkpoint is on Hugging Face and calls the system completely open, and it discloses unusually detailed training recipes (loss weights, data mixture, augmentation probabilities). However, the serving stack relies on a custom vLLM fork with no stated release path, the TTS component is cited to an anonymous manuscript under review, and the tool-calling data pipeline is only described procedurally — so evaluation-level reproduction is plausible, but full retraining is not.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 (architecture overview and tool-channel state machine) appear only as garbled layout text; visual structure could not be verified.
- Section 5 and Tables 1-2 use the name 'V-Model' throughout; this review assumes it denotes the same system as the title's NemotronLabs VoiceChat.
- Reference [20] (VoiceChat-TTS) is an anonymous manuscript under review, so its standalone evaluation protocol could not be checked.
- Table 1's Gemini Live 2.0 row formatting had to be partially inferred from surrounding prose due to extraction artifacts.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.