Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech
Chibuzor Okocha, Christan Earl Grant
cs.CL · cs.AI
Quick verdict
Claims WER hides code-switch failure: 11-system English–Yoruba benchmark; core dissociation convincing, but single corpus and small-n stats limit generality.
Summary
This paper benchmarks eleven zero-shot speech systems — six dedicated ASR models (Parakeet-TDT-0.6B-v2, Qwen3-ASR-1.7B, Canary-Qwen-2.5B, Whisper-large-v3 in forced and auto modes, Granite-Speech-4.1-2B) and five audio language models (Kimi-Audio-7B, Audio Flamingo 2/3, Qwen2-Audio, Qwen2.5-Omni) — on English–Yoruba code-switched speech from the AFRICODESWITCH validation partition, using a deterministic 2,000-utterance manifest and shared scoring. Beyond WER it introduces switch-localized diagnostics: SETER (error on the first token of each language switch), SPER@k windowed rates, language-specific error rates, and diacritic-insensitive WER. The central result is a dissociation: Parakeet has the best WER (66.1%) yet statistically ties Kimi-Audio (66.2%, bootstrap p=0.76), while Kimi-Audio wins every switch-localized metric significantly (p<0.001). Yoruba recognition collapses (token error ≥96.6% across faithful systems) versus 33–46% English error, and switch-token error is 0.95–1.00 entering Yoruba versus 0.40–0.53 entering English. Three audio LMs exceed 100% WER through prompt leakage and verbosity that prompting largely fixes (Qwen2-Audio leakage 33.7%→0.2%, WER 135.6%→103.3%). Diacritic stripping moves strong-system WER by ≤0.3 points, so Yoruba coverage, not tone marks, is the bottleneck. Limitations: a single uncited corpus whose reference language tags the new metrics inherit; rank correlations over ~11 systems; the human-annotation taxonomy still in progress; planned systems (Qwen3-Omni, a Cohere model) dropped.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Aggregate WER is decoupled from code-switching fidelity: the WER-best ASR model (Parakeet, 66.1%) statistically ties Kimi-Audio (66.2%, bootstrap p=0.76) but is significantly worse on every switch-localized metric (p<0.001, Table III).
- Yoruba token recognition collapses across faithful systems (YO-ER ≥0.966) while English error is 33–46%, and switch-token error is 0.95–1.00 entering Yoruba versus 0.40–0.53 entering English (Fig. 1b).
- WER correlates strongly with English token error (ρ=0.74) but is uncorrelated with Yoruba error (ρ=−0.10), so corpus-level WER cannot in principle reflect embedded-language competence in asymmetric code-switching.
- Diacritics are currently a second-order concern: stripping tone/sub-dot marks moves strong-system WER by at most 0.30 points, so base Yoruba coverage is the binding constraint (Section VI-E).
- Generative audio LMs fail as verbatim transcribers through prompt-dependent leakage and verbosity: a direct prompt cuts Qwen2-Audio's leakage from 33.7% to 0.2% and WER from 135.6% to 103.3% on cap1000 (Table V).
Strengths
Cleanly demonstrated core dissociation
A paired utterance-level bootstrap on the shared cap2000 manifest (Table III) shows the Parakeet–Kimi-Audio WER gap is null (Δ=+0.13 pts, p=0.76, CI [−0.67,+1.12]) while Kimi-Audio wins SETER (−2.09), SPER@1 (−2.20) and SPER@3 (−2.02) at p<0.001; the Canary-Qwen contrast (significantly worse WER, indistinguishable SETER/SPER at p>0.12) independently corroborates the thesis.
Principled metric suite
SETER and SPER@k are well-defined positional instantiations of a single masked error rate (Eq. 1), honestly positioned as boundary-localized complements to PIER's language-membership view; adding WERdi and per-language error rates yields a reusable diagnostic panel rather than a one-off score.
Actionable prompt-confound finding
Honest negative result on diacritics
Internal consistency verification
Weaknesses
Single, uncited corpus underpins everything
Underpowered and inconsistent rank statistics
No multiplicity control; most contrasts untested
Human validation of generative-failure claims unfinished
Extreme error regime and dropped systems
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an evaluation-only benchmark study: eleven zero-shot systems — six ASR models (Parakeet-TDT-0.6B-v2, Qwen3-ASR-1.7B, Canary-Qwen-2.5B, Whisper-large-v3 in forced-English and auto-detect modes, Granite-Speech-4.1-2B) and five audio LMs (Kimi-Audio-7B-Instruct, Audio Flamingo 2/3, Qwen2-Audio-7B-Instruct, Qwen2.5-Omni-7B) — transcribe English–Yoruba code-switched speech from the AFRICODESWITCH validation partition under a deterministic 2,000-utterance manifest (cap2000) and one shared scoring pipeline. The headline claims: (1) aggregate WER conceals code-switching failure — the WER-best system statistically ties Kimi-Audio yet loses every switch-localized metric; (2) Yoruba recognition collapses (YO-ER ≥ 0.966) while English error is 33–46%; (3) errors concentrate at switches into Yoruba (0.95–1.00) versus into English (0.40–0.53); (4) diacritics are second-order; (5) generative audio LMs fail as transcribers via prompt-dependent translation, verbosity, and leakage. The durable contribution is the metric suite (SETER, SPER@k, WERdi) plus stated releases of manifests and scoring scripts.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SETER in speech recognition evaluation?
SETER (Switch-Entry Token Error Rate) is the fraction of reference tokens that begin a new language run in code-switched speech — the exact point where the speaker switches language — that the system failed to transcribe correctly. It complements WER by scoring only switch-boundary tokens; the related SPER@k widens the scored window to k tokens around each switch point (SPER@0 equals SETER).
Why is WER considered misleading for code-switched ASR?
Because one (usually higher-resource) language supplies most of the words, aggregate WER is dominated by accuracy on that matrix language. In this study WER tracked English error closely (Spearman ρ=0.74) but was uncorrelated with Yoruba error (ρ=−0.10): the best-WER system (66.1%) statistically tied another system on WER while being significantly worse on every switch-localized metric.
Which ASR models and audio language models were tested on English–Yoruba code-switched speech?
Eleven zero-shot systems on a fixed 2,000-utterance set from the AFRICODESWITCH validation partition: six ASR models (Parakeet-TDT-0.6B-v2, Qwen3-ASR-1.7B, Canary-Qwen-2.5B, Whisper-large-v3 with forced English and with auto language detection, Granite-Speech-4.1-2B) and five audio LMs (Kimi-Audio-7B-Instruct, Audio Flamingo 2 and 3, Qwen2-Audio-7B-Instruct, Qwen2.5-Omni-7B).
Are the paper's code, metrics, and dataset publicly available?
The authors state they release evaluation manifests, metric implementations, and evaluation scripts, but no repository URL appears in the paper text. The AFRICODESWITCH corpus itself is not cited with a source, license, or availability statement, so independent reproduction currently depends on obtaining that corpus.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal content (Fig. 1 panels, Fig. 2 domain bars) was extracted as jumbled inline axis labels and numbers; plotted curve shapes and per-point values beyond those quoted in the text could not be verified.
- No repository URL for the released manifests/scripts appears anywhere in the extracted text, so the release claim could not be checked against an external source (network access disabled for this task).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.