Kraken: LLM-based Speech-to-Speech Translation via Low-bitrate VQ and Dual-path Source Conditioning
Hayato Futami, Hassan Shahmohammadi, Tushar Dhyani, Alkis Koudounas, Raphaël Lafargue, Yosuke Kashiwagi, Quentin Jodelet, Emiru Tsunoo
cs.CL · cs.SD
Quick verdict
Kraken beats SeamlessM4T-Lv2 and Qwen2.5-Omni on X-En S2ST via 325bps VQ and source-conditioned vocoder; solid engineering, significance untested.
Summary
The paper presents Kraken, a speech-to-speech translation (S2ST) system built on Qwen3-8B and fully fine-tuned on 150k hours of multilingual, multitask speech data (ASR, S2TT, S2ST, MT, TTS) spanning 10 languages. To avoid the difficult prediction of high-bitrate speech tokens, it uses a single-layer VQ codec (8192-entry codebook, 13 bits at 25 Hz, 325 bps) trained to reconstruct layer-20 W2v-BERT 2.0 SSL embeddings. A GAN vocoder, Autowave-X (UniCATS + Vocos), converts tokens to waveforms and is additionally conditioned on layer-8 source embeddings, enabling speaker/prosody transfer without speaker-aligned training data; a WaveFit-based iterative refiner optionally improves audio quality. Chain-of-thought prompting performs ASR and S2TT before speech token generation. On FLEURS X-En, Kraken (9.4B) attains 32.3 average ASR-BLEU versus 32.1 for SeamlessM4T-Large v2 and 25.1 for Qwen2.5-Omni, trailing only the 30B Qwen3-Omni (33.6); on in-domain CVSS it leads all baselines (43.5). Objective speaker (SSim 0.75), emotion (0.77) and prosody (AutoPCP 2.48) similarity are the best among compared systems, and human evaluation on 25 MELD-ST clips (3 annotators) favors Kraken on prosody, speaker and emotion match, though UTMOS (3.13) trails Qwen-Omni models. Evaluation is limited to X-to-English; no code, models or demos will be released, and no significance testing is reported.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A single-layer low-bitrate VQ (325 bps) trained to reconstruct W2v-BERT 2.0 SSL features provides a sufficient target-speech representation for LLM-based S2ST.
- Conditioning both the LLM and the Autowave-X decoder on source speech enables speaker and prosody transfer from speaker-mismatched S2ST training data.
- Kraken (9.4B) outperforms SeamlessM4T-Large v2 and Qwen2.5-Omni in X-En ASR-BLEU on FLEURS and CVSS, and surpasses the 30B Qwen3-Omni on in-domain CVSS.
- Kraken achieves the best objective speaker, emotion and prosody similarity among compared systems, corroborated by human evaluation on MELD-ST dubbing.
Strengths
Compact single-layer token space fits standard LLM decoding
325 bps (13 bits at 25 Hz, one 8192-entry codebook) versus ~2.2 kbps RVQ in Qwen3-Omni/Hibiki and 664 bps Seamless units (Section 2.2), avoiding depth transformers or multi-token prediction modules.
Dual-path source conditioning is ablation-backed
Table 4: removing decoder conditioning drops SSim 0.75 to 0.65 and APCP 2.48 to 2.44; removing LLM conditioning yields SSim 0.73 and APCP 2.38 — both paths measurably contribute to source transfer.
Strong expressiveness transfer at 9.4B scale
Transparent negative results
In-domain efficiency against a 3x larger model
Weaknesses
No statistical significance testing anywhere
Audio quality trails competitors with a confounded explanation
Strongest headline claim is in-domain
Uncontrolled baseline comparison
Effectively unverifiable
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Kraken is an offline speech-to-speech translation (S2ST) system covering 10 source languages translated into English. It augments a fully fine-tuned Qwen3-8B LLM with a frozen, ASR-tuned W2v-BERT 2.0 encoder (continuous features downsampled to 12.5 Hz) and a decoder that turns 325-bps VQ tokens into waveforms. Four central claims: (1) low-bitrate single-layer tokens make S2ST tractable for a standard LLM; (2) dual-path source conditioning enables speaker and prosody transfer from speaker-mismatched S2ST data; (3) translation quality exceeds SeamlessM4T-Large v2 and Qwen2.5-Omni on FLEURS and CVSS and surpasses the 30B Qwen3-Omni on CVSS; (4) speaker, emotion and prosody transfer are the best among compared systems.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Kraken speech translation model?
Kraken is a speech-to-speech translation system from Sony built on the Qwen3-8B language model. It takes speech in one of 10 languages and produces translated English speech: the LLM generates compact 325-bits-per-second speech tokens, which a GAN vocoder called Autowave-X converts to audio while also receiving the source speech to preserve the original speaker's voice and prosody.
How does Kraken preserve the speaker's voice in translation?
It uses dual-path source conditioning: the LLM receives continuous source-speech features as input, and the Autowave-X vocoder is additionally conditioned on layer-8 self-supervised speech embeddings, which carry speaker identity. This lets it transfer the source speaker and prosody even when the S2ST training pairs had mismatched speakers.
Is Kraken's code or model publicly available?
No. The Ethics Statement says the authors will not release demos, code, or models publicly, citing voice-cloning and deepfake misuse risks; the system also relies on internal training data, so results cannot currently be independently reproduced.
How does Kraken compare to SeamlessM4T and Qwen3-Omni?
On FLEURS X-to-English tests it slightly beats SeamlessM4T-Large v2 (32.3 vs 32.1 ASR-BLEU) and clearly beats Qwen2.5-Omni (25.1), but trails the much larger 30B Qwen3-Omni (33.6). On the in-domain CVSS benchmark it tops all three (43.5 vs 40.9). It transfers speakers and prosody best of the compared systems, but its audio naturalness (UTMOS 3.13) is rated below the Qwen-Omni models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure contents (e.g., the task/language distributions in Figure 5) are not recoverable from text extraction; all numeric tables extracted cleanly and completely.
- PDF extraction interleaves fragments of the Figure 1 caption with body text near the Introduction; this does not affect any numbers cited in this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.