BuzzASR: A Swarm of 100+ Monolingual Speech Recognition Models
Shivam Singh, Aditya Yadavalli, Catherine Arnett, Alex Warstadt
cs.CL
Quick verdict
102 monolingual Whisper fine-tunes beat Whisper-large-v3 on 77/102 languages (~2.8× mean CER cut); solid gains, but train/test domain match tempers SOTA claims.
Summary
The authors release BuzzASR: 102 language-specific ASR models fine-tuned from Whisper-large-v3 on FLEURS and Common Voice training data (capped at 100/90 hours per language). Beyond simple whole-model fine-tuning (SFT), a full fine-tuning (FFT) pipeline replaces Whisper's multilingual BPE with a per-language BPE (warm-start initialized embeddings) and mixes text-only language-modeling batches with ASR batches 50/50. On the combined FLEURS+Common Voice test sets, the best per-language model beats Whisper-large-v3 zero-shot on 77/102 languages (median CER reduction 2.2×, mean 2.84×); against Omnilingual-1B/7B, MMS, Qwen3-ASR, and Cohere Transcribe it posts the lowest CER on 27/102 languages, though Omni-7B wins most head-to-heads (BuzzASR beats it on only 42). Adapted tokenizers raise median compression from 2.38 to 5.04 characters/token and cut decoder latency roughly 1.6×. Key limitations, mostly self-acknowledged: train and test data share domains, and a 17-language out-of-domain probe shows part of the gain reflects that match; fine-tuning severely degrades some languages (Finnish CER 1.89→29.28); per-language best-of-multiple-runs reporting lacks variance estimates; and the abstract's 21.7× tokenizer gain comes from Kamba, a language the paper itself flags as anomalous and excludes from aggregates. Models, code, and detailed results are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The best BuzzASR model per language outperforms Whisper-large-v3 zero-shot on 77 of 102 languages, with median CER reduction 2.2× and mean 2.84× on the combined FLEURS+Common Voice test set.
- BuzzASR achieves the lowest CER among the compared open-source systems on 27 of 102 languages, with Omnilingual-7B the strongest baseline overall (BuzzASR wins only 42 head-to-heads by CER).
- Per-language BPE tokenizer replacement with warm-start initialization raises median compression from 2.38 to 5.04 characters per token and cuts decoder latency by roughly 1.6×.
- Full fine-tuning (tokenizer replacement plus 50/50 text/ASR multitask) helps most where Whisper is weakest, while simple fine-tuning suits languages Whisper already handles well.
Strengths
Scale with systematic comparison
Fine-tunes across 102 languages on a fixed A/B/C config grid (Table 3), evaluated head-to-head against five modern baselines including Omnilingual-1B/7B, with per-strategy win rates in Table 1.
Exemplary ablations
Warm-start vs random vs extended tokenizer init (Section D: median CER 7.4 vs 15.1/15.7 on 17 core languages); multitask without tokenizer replacement (Section G); bilingual vs monolingual (Section F: monolingual wins by 1.24–1.25× median, related or not).
Honest self-diagnosis of domain matching
Well-instrumented tokenizer analysis
Full artifact release
Weaknesses
Train/test domain overlap
Selection and bookkeeping inconsistencies
Abstract overstates tokenizer gains
Unfair scoring for two baselines
Regressions understated in prose
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
BuzzASR (arXiv 2609.09554) is a release-plus-evaluation paper: 102 language-specific ASR models fine-tuned from Whisper-large-v3 on FLEURS and Common Voice v25 training data (100h/90h caps per language; test data capped at 2,000 utterances for 36 of 77 CV-covered languages). Two adaptation strategies are compared: simple fine-tuning (SFT, whole model on ASR data) and full fine-tuning (FFT), combining per-language BPE tokenizer replacement (51,865-token vocab; warm-start embedding and unembedding initialization as averages of original subword embeddings) with multitask training mixing text-only fish-food batches (500k lines) and ASR batches at α=0.5 (Sections 3–4). Headline claims: wins on 77/102 languages over Whisper-large-v3 zero-shot (median CER reduction 2.2×, mean 2.84×, Section 5.1); lowest CER among five compared open systems on 27/102 (Table 1); tokenizer compression median 2.38→5.04 chars/token and ~1.6× decoder latency cut (Section 5.2, Table 5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is BuzzASR?
BuzzASR is a released collection of 102 monolingual speech recognition models, each fine-tuned from Whisper-large-v3 on the FLEURS and Common Voice training data for one language. On a combined FLEURS+Common Voice test set, the best model per language beats Whisper-large-v3 zero-shot on 77 of 102 languages and posts the lowest CER among the compared open-source systems on 27 languages. Models, code, and per-language results are on the project page (lemn-lab.github.io/buzz-asr).
How much better is BuzzASR than Whisper?
Per the paper, the best BuzzASR model per language beats Whisper-large-v3 on 77 of 102 languages, cutting normalized character error rate by a median factor of 2.2 (mean 2.84). Gains concentrate where Whisper is weakest, including more than 6× CER cuts for ten languages led by Amharic, Maltese, and Sorani Kurdish — while for languages like Finnish, Tamil, and Thai fine-tuning made results substantially worse.
Does BuzzASR beat larger models like Omnilingual-7B?
No. The paper reports Omnilingual-7B as the strongest baseline, winning most head-to-head comparisons — BuzzASR beats it on only 42 of 102 languages by CER. BuzzASR fares better against the size-comparable Omni-1B (wins on 60 of 102) and is positioned as a competitive option when a 7B model is too large to deploy.
Why replace the tokenizer in an ASR model?
Whisper's multilingual BPE was trained mostly on English, so it fragments non-Latin scripts into many tokens, lengthening sequences and slowing decoding. BuzzASR trains a per-language BPE and initializes new token embeddings from the averaged embeddings of their original subword pieces (warm start). Median compression rises from 2.38 to 5.04 characters per token with roughly 1.6× faster decoding; the abstract's 21.7× headline gain comes from one language (Kamba) the paper itself flags as anomalous and excludes from its aggregate statistics.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (1, 3–10) appear only as extracted axis-label fragments; the plotted distributions could not be inspected, so figure-based claims were checked against the tables and text instead.
- A few table cells show minor extraction artifacts around decimal points (e.g., German CER rendered as '6 09'); individual values cited in this review were cross-checked against surrounding rows.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.