ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding

Summary

ZipCodec is a streaming neural speech codec operating at 6.25 Hz frame rate and 0.80 kbps with 160 ms theoretical latency, claimed to be the first streaming codec jointly preserving semantic and acoustic information below 12.5 Hz. Built on FocalCodec-Stream, it uses a causal log-mel frontend, 16-frame temporal patching, an ErfFormer backbone (LLaMA-style, with no normalization layers or positional encodings), scalar spherical quantization (64 dimensions, 4 levels), and a latency-aware streaming Vocos decoder. The pipeline is trained by distilling WavLM layer-6 representations on roughly 94,000 hours of English speech (LibriLight, VoxPopuli, GigaSpeech) with WavLM-style noise and overlap augmentation; the waveform decoder is trained separately on LibriTTS-100. On speech resynthesis (LibriSpeech test-clean and an MLS subset), ZipCodec improves over FocalCodec-Stream (e.g., English dWER 2.83 vs 3.68) and approaches the non-streaming FocalCodec@50. Across five discriminative and two generative downstream tasks it is the best or second-best streaming codec, and it sustains real-time single-stream inference on a consumer CPU (RTF 1.33, p99 step latency 124.68 ms). Main limitations: no ablations isolating the many bundled changes, no statistical significance testing for small margins, no human listening tests, English-only training despite multilingual evaluation, and a model/data scale far above baselines that confounds comparisons.