Multimodal Duplex Interaction Agent
Orantqing, Shengpeng Ji, Junlong Tong, Jialong Zuo, Dongjie Fu, Di Cao, Yangzhuo Li, Shangda Wu, et al.
eess.AS · cs.AI · cs.LG · cs.MM · cs.SD
Quick verdict
Gander adds an agent loop to full-duplex speech interaction; strongest turn-taking (100%/8.0%) but last-place task accuracy on its main benchmark.
Summary
Gander is a full-duplex multimodal interaction agent built on MiniCPM-o 4.5 (9B) by Tencent's Hunyuan Speech Team and university collaborators. A realtime 'Cerebellum' handles spoken conversation using a streaming Thinker-Talker design: audio-visual inputs and text outputs are flattened into one-second chunks, and the model predicts a listen/speak/interrupt control token per chunk within a 128-chunk (about two-minute) sliding window. A training-free 'Brain' (Codex driven by GPT-5.6) executes long-horizon tasks, coordinated through an orchestration runtime via task_start/task_send/task_resolve tool calls. Training uses roughly 2.7 million synthesized samples spanning speech interaction, audio-visual interaction, agentic trajectories, and robustness/negative data. On Full-Duplex-Bench v3, Gander posts the strongest turn-taking profile (100.0 take-turn, 8.0 interrupt) yet ranks last among seven systems on all four accuracy metrics (Pass@1 0.400 versus 0.600 for GPT-Realtime); a back-brain-only condition leads tool selection at 0.934. On SpokenQA/VoiceBench it leads the full-duplex group on knowledge subsets (75.60 and 59.30) but trails on open-ended and accented-speech subsets. Multimodal understanding stays within 1.67 points of its base model on Daily-Omni (78.53) but drops 6.08 points on WorldSense. Main limitations: undisclosed human-evaluation protocol, absence of significance testing, proprietary back-brain dependency, and an unexplained end-to-end accuracy gap.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Modeling interaction state intrinsically via per-chunk listen/speak/interrupt control tokens yields stable full-duplex turn-taking without external VAD (Take-turn 100.0, Interrupt 8.0 on Full-Duplex-Bench v3).
- Decoupling a 9B realtime Cerebellum from a training-free Brain preserves strong tool-selection at the execution tier (back-brain-only ToolSel 0.934, the highest in Table 3).
- A frame-synchronous streaming model remains competitive on knowledge-oriented spoken QA (75.60 Llama Questions / 59.30 Web Questions, second among nine systems).
- Interaction-oriented post-training retains multimodal understanding within 1.67 points of the base model on Daily-Omni (78.53 vs 80.20), though WorldSense falls 6.08 points.
Strengths
Honest negative reporting in the main benchmark
Table 3 shows Gander last among seven systems on all four accuracy metrics (ToolSel 0.759, ArgAcc 0.503, RespQual 0.490, Pass@1 0.400) and Section 5.2 states this plainly; the back-brain-only row (ToolSel 0.934, Pass@1 0.520) cleanly isolates execution-tier capability from end-to-end degradation.
Above-average evaluation protocol disclosure
Section 5.1: single checkpoint, byte-identical training system prompt, unmodified released scoring scripts, scoring over ASR transcripts of synthesized speech (so TTS/ASR errors are included), unextractable answers counted wrong, greedy decoding making Tables 5-6 deterministic.
Unusually well-specified duplex data pipeline
Coherent intrinsic-interaction mechanism
Weaknesses
End-to-end accuracy gap is unresolved
Human evaluation is unverifiable
No statistical treatment
Benchmark protocol patch not ablated
Understanding regression with speculative attribution
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Gander (arXiv 2609.08977v3; Tencent Hunyuan Speech Team with Zhejiang University, SJTU, CUHK, NTU) is a system paper for a native full-duplex multimodal interaction agent built on MiniCPM-o 4.5 (9B). Three claims anchor the work: (1) interaction state belongs inside the model — a streaming chunk formulation predicts a listen/speak/interrupt control token each second, removing external VAD; (2) a Cerebellum-Brain split couples realtime conversation with asynchronous agentic execution, the Brain being training-free (a Codex worker driven by GPT-5.6) and plug-and-play; (3) the released system (weights, code, data claimed) is competitive across dialogue, understanding, and tool-assisted tasks. Claim (3) is the weakest link: the paper's own Table 3 contradicts a strong reading of it.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Gander and who built it?
Gander is a full-duplex multimodal voice agent from Tencent's Hunyuan Speech Team with Zhejiang University, SJTU, CUHK and NTU. It listens and speaks simultaneously while delegating complex tasks (code, files, retrieval) to a background coding agent (Codex driven by GPT-5.6), built on MiniCPM-o 4.5 (9B parameters).
Does Gander beat GPT-Realtime?
Not overall. On Full-Duplex-Bench v3, Gander has the best turn-taking profile (takes the turn appropriately in 100% of scenarios and interrupts prematurely in 8.0% of turns vs 13.5% for GPT-Realtime), but it scores lower on every accuracy metric (Pass@1 0.400 vs 0.600). A back-brain-only variant reaches the best tool-selection score of all systems (0.934).
Is Gander's code and model public?
The paper states that model weights, code, and data are released. However, its 'Brain' execution tier depends on the proprietary Codex/GPT-5.6 agent and benchmark scoring uses a GPT-4o judge, so exact end-to-end numbers are hard to replicate; training compute is not reported.
What data was Gander trained on?
About 2.7 million synthesized samples in four families: speech interaction (roughly 37%, including 260.8K full-duplex dialogues with injected interruptions and backchannels carrying explicit timing supervision), audio-visual interaction (1.1M samples), agentic interaction trajectories (~360K), and robustness/negative data teaching the model to stay silent or ignore distractions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 are extracted as unordered text fragments, so the architectural diagrams could be assessed only via captions and body prose; all tables and numbers cited in this review are present in the extracted text, which runs complete through the references with no truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.