The Hidden Cost of Digits: Number Normalization and WER in ASR Systems
Stanisław Kacprzak, Mieszko Fraś
eess.AS · cs.CL
Quick verdict
Shows unnormalized numbers distort multilingual ASR WER by up to ~2pp, enough to flip system rankings; solid empirical work, but lacks significance testing.
Summary
This empirical evaluation study quantifies how neglecting number normalization distorts word error rate (WER) comparisons in multilingual ASR. Because large models like Whisper emit Arabic numerals while others (e.g., Canary) transcribe numbers verbatim, mixed conventions inflate or deflate WER unpredictably. The authors test Whisper large-v3, a prompt-induced verbatim variant, OWSM v4 medium, and Canary 1B/1B-v2 on VoxPopuli test sets (English, Polish, German, French, Spanish; roughly 1,500-1,970 recordings each) and a 128-recording Polish Parliamentary corpus, comparing four transcript states: verbatim, digit, and number-normalized versions of each. They contribute a rule-based PolishNumberNormalizer, built on OpenAI's EnglishNumberNormalizer with Polish inflection, date, and decimal handling, released with unit tests. Table 3 shows digit-vs-verbatim transcript WER of 0.98% (English VoxPopuli), 1.28% (Polish VoxPopuli), and 2.19% (Polish Parliamentary), dropping to 0.31%, 0.30%, and 0.00% when both sides are normalized. Table 4 shows full normalization yields the lowest WER for every system-dataset pair; gains reach about 2 percentage points on the parliamentary data (e.g., Canary-v2 17.49→15.45) and up to about 0.9 points on VoxPopuli, often exceeding gaps between top benchmark systems. Prompted Whisper-IP outperforms standard Whisper under verbatim references (8.38 vs 8.46) despite being worse overall — a concrete ranking flip. Limitations: no significance testing, a small parliamentary set, heterogeneous normalizers across languages, and parliamentary-domain data only.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Lack of number normalization can distort WER by up to ~2 percentage points on number-dense Polish parliamentary data (Canary-v2 17.49→15.45, Whisper 15.34→13.32, Table 4) and up to ~0.9 points on VoxPopuli.
- The lowest WER for every system-transcript-dataset combination tested is obtained when both the reference and the ASR output are number normalized (Table 4, bolded entries).
- Normalization neglect can invert system rankings: prompted Whisper-IP beats standard Whisper under verbatim references (8.38 vs 8.46 WER, English VoxPopuli) despite being ~0.5pp worse in normalized scenarios.
- Polish's high inflection (1984 = 4 words vs 6 in French, 5 in Spanish, 3 in English, 1 in German, Table 2) makes number normalization hard and motivates the new rule-based PolishNumberNormalizer.
Strengths
Quantifies a real, widely overlooked evaluation bias
Section 4.2 demonstrates a concrete ranking inversion: prompted Whisper-IP scores 8.38% WER vs standard Whisper 8.46% on English VoxPopuli verbatim references despite being ~0.5pp worse once normalized — exactly the misleading comparison the paper warns about.
Clean factorial design isolating normalization effects
The four reference states (V, N(V), D, N(D)) in Table 3 separate aligned-scenario gains from single-side mismatch penalties (worst case 1.33% English VoxPopuli vs 0.31% when both sides are normalized), a more careful decomposition than typical WER reports.
New, validated public tool for an underserved language
Transparent about scope and tooling limits
Weaknesses
No uncertainty quantification anywhere
Cross-language normalizer heterogeneity confounds Figure 1
Parliamentary-only domain and smallest set drives the headline number
Manual references and post-hoc exclusions lack protocol detail
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a narrow but consequential question: how much does unnormalized number formatting bias WER when comparing multilingual ASR systems with different output conventions (digits vs. words)? Three claims are made. First, the distortion reaches more than 2 percentage points on number-dense Polish parliamentary speech (Table 4: Canary-v2 17.49→15.45; Whisper 15.34→13.32) and up to ~0.9 points on VoxPopuli (e.g., Polish Whisper 7.85→6.97). Second, full normalization on both reference and hypothesis sides yields the lowest WER for every system-dataset combination tested (Table 4 bolded entries). Third, normalization neglect can invert rankings: prompted Whisper-IP scores 8.38% vs standard Whisper's 8.46% on English VoxPopuli verbatim references despite being ~0.5pp worse once normalized. The work is honestly framed as a complement to Huber & Waibel [15] and to prior multilingual-normalization critiques [5,6], and the Polish tool addresses a real gap in publicly available normalizers for inflectional languages.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How much can number normalization change ASR word error rate?
In this paper, normalizing both the reference transcript and the ASR output reduced WER by roughly 0.3-0.9 percentage points on VoxPopuli across five languages, and by about 2 percentage points on a number-dense Polish parliamentary corpus — for example, Canary-1B-v2 dropped from 17.49% to 15.45% WER and Whisper large-v3 from 15.34% to 13.32%. These gains often exceed the gaps separating top systems in multilingual benchmarks.
Why do different ASR systems write numbers differently?
It follows from training-data conventions: Whisper and OWSM mostly output Arabic digits (e.g., '123'), Canary 1B spells numbers out as words ('one hundred twenty-three'), and Canary 1B-v2 mostly spells them out. When you compare such systems against a reference in the other convention without normalization, whichever system mismatches the reference gets unfairly penalized in WER.
What is the PolishNumberNormalizer?
A free, rule-based tool the authors built on top of OpenAI's EnglishNumberNormalizer. It converts Polish spelled-out numbers into digits, handling Polish-specific conventions such as decimal commas, ordinal inflections, fractions, dates and times, and it runs before punctuation removal to avoid ambiguity. It ships with unit tests and reduced the WER between digit and verbatim parliamentary references to 0.00% after normalization.
Does Whisper's built-in text normalization work for non-English languages?
Only partially. Whisper's shared code applies basic cleanup (lowercasing, punctuation and symbol removal) to all languages, but the English-specific transformations — including number normalization via EnglishNumberNormalizer — are not applied outside English. This paper quantifies exactly that gap: without language-specific number normalization, WER comparisons in Polish, German, French, and Spanish can be off by up to roughly 2 percentage points.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (per-language absolute WER-reduction gains) is referenced but its values are not recoverable from the extracted text.
- Table 4 arrived flattened; the raw-vs-N(·) column pairing was reconstructed from the header and Section 4.2 prose and is internally consistent with the cited numbers.
- Footnote 1's repository URL was truncated in extraction; code availability is inferred from the footnote text rather than a verified link.
- Minor extraction artifacts (e.g., 'overoptimisticcall' in Section 1) indicate some OCR noise, but all tables and headline claims remained verifiable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.