Temporal Taxation Compounds Under Post-Training Compression of Whisper Models
Srishti Ginjala, Eric Fosler-Lussier, Christopher W. Myers, Srinivasan Parthasarathy
cs.CL · cs.SD
Quick verdict
Claims ASR compression shifts error burden onto marginalized speakers; rigorous single-family audit, but limited to data-free 4-bit recipes and read speech.
Summary
This paper audits whether post-training weight compression — quantization, pruning, and distillation — redistributes error burden across demographic groups in Whisper speech-recognition models, evaluated on Fair-Speech, Common Voice 25, and AfriSpeech-200. Operationalizing the 'temporal taxation' construct of Choi and Choi (2025) as a linear rescaling of per-group word error rate into correction-time units, the authors report that 50% Wanda pruning of Whisper-large-v3 more than doubles (+111%) the Black/AA-versus-Asian worst-versus-best differential on Fair-Speech, from 30.14 to 63.73 seconds of correction per minute of speech at an assumed five seconds per error. The relative increase is invariant to that cost assumption, survives an SNR audio-quality control, and beam-5 decoding still leaves +86%. At edge model size, INT4 HQQ quantization multiplies catastrophic loop rates on West African accents by five to seven, while INT8 is nearly fairness-neutral. Contrary to long-tail predictions from vision and multilingual NLP, distillation narrows gaps in 21 of 27 settings, with widening confined to one teacher-student pair. Limitations include a single model family (the Granite probe covers INT8 only), English read speech, one implementation per method (GPTQ/AWQ/SparseGPT excluded), single checkpoints, and a partial SNR proxy. Statistical treatment is careful: paired permutation tests, BH-FDR correction, bootstrap intervals, and threshold sensitivity analyses.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- 50% Wanda pruning of Whisper-large-v3 more than doubles the Black/AA-vs-Asian temporal-taxation differential on Fair-Speech (+111%), invariant to per-error cost, robust to an SNR control, and still +86% under beam-5 decoding.
- At edge model size, INT4 HQQ quantization multiplies catastrophic loop rates on West African accents by 5–7x, while INT8 is nearly fairness-neutral (7 BH-significant deltas out of 427).
- Distillation narrows demographic gaps in 21 of 27 evaluated settings, with widening concentrated on the whisper-medium → distil-medium.en pair on Fair-Speech ethnicity.
- Full-precision model rankings by max-min ratio survive compression (Kendall's tau = 1.000 in 11 of 12 cells), so single-snapshot audits capture ordering but not deployment-time magnitude of the disparity.
Strengths
Fills a genuine gap with a well-scoped first
First systematic pruning x ASR-fairness matrix and first sub-8-bit fairness audit on English dialect/accent axes; the GPTQ/AWQ/SparseGPT exclusions are documented with concrete library blockers (Appendix E) rather than waved away.
Statistical protocol well above field norm
Paired permutation tests (B=2,000) with BH-FDR within cells, B=200 bootstrap CIs, Kendall's tau rank stability (tau=1.000 in 11/12 cells, Appendix A), pre-registered loop thresholds with a three-setting sensitivity analysis (Appendix D, Tables 3–4) showing Kanuri/Yoruba/Hausa significant at every threshold.
Confound control that strengthens the result
Headline claim is assumption-robust
Honest counter-expectation reporting
Weaknesses
Headline effects are Whisper-only
Practical-method coverage gap
Aggregate operating point unreported for the headline cell
Ratio metric fragility and small strata
Single checkpoints, unmeasured run-to-run variance
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a deployment-shaped question: ASR fairness audits are performed at full precision, but shipped systems are compressed. Across eight Whisper-family models (five backbones, 39M–1.55B; three Distil-Whisper students), four precisions (FP16, INT8, INT4-NF4, INT4-HQQ), 50% Wanda pruning, and three English benchmarks (Fair-Speech, Common Voice 25, AfriSpeech-200), it claims: (1) 50% Wanda pruning of whisper-large-v3 more than doubles the Black/AA-vs-Asian worst-vs-best temporal-taxation differential on Fair-Speech (30.14 → 63.73 s of correction per minute of speech at 5 s/error; +111%, falling to +86% under beam-5, Section 4.1/5.4); (2) at edge sizes, INT4 HQQ multiplies catastrophic loop rates on West African accents 5–7x (Kanuri 4.55% → 30.30%, 6.67x, Section 5.2); (3) INT8 is nearly fairness-neutral (7 of 427 significant deltas, Appendix A); (4) distillation narrows gaps in 21/27 cells, widening confined to whisper-medium → distil-medium.en on Fair-Speech ethnicity (Section 6, Figure 5); (5) full-precision MMR rankings survive compression (Kendall's tau = 1.000 in 11/12 cells, Appendix A). The authors correctly disclaim the temporal-taxation arithmetic itself as a contribution; its value is the cost-invariant relative measure (Section 3.4, Appendix B).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'temporal taxation' in speech recognition?
Temporal taxation is a fairness framing from Choi and Choi (2025) that this paper turns into a number: per-group word error rate multiplied by an assumed correction cost per error (e.g., 5 seconds) and speaking rate, yielding estimated seconds of transcript-repair labor per minute of speech. Because the conversion is linear, relative changes (like the paper's +111% pruning effect) do not depend on the assumed per-error cost; only absolute values do.
Does compressing Whisper models make ASR less fair for accented or minority speakers?
It depends on method and model size. INT8 quantization is nearly fairness-neutral in this study. At edge sizes, INT4 quantization (especially the HQQ recipe) raises error rates and multiplies catastrophic repetition loops most on West African accents (5–7x). Distillation mostly narrows demographic gaps. The worst result is 50% Wanda pruning of the largest model, which more than doubles the worst-vs-best group gap on Fair-Speech.
What datasets were used to test compression fairness in Whisper?
Three English, read-speech benchmarks: Fair-Speech (26,471 voice-assistant prompts from 593 US speakers with self-identified ethnicity, age, gender), Common Voice 25 English test split with accent labels for seven groups, and AfriSpeech-200 (6,318 utterances across 44 African accents). LibriSpeech test-clean served only as an aggregate reference and its training split as pruning calibration data.
Is code or data available for this paper?
As of this v1 preprint, no — the paper states that code, configurations, and result tables 'will be released on publication.' The methods are described in enough detail (pinned environment, deterministic decoding, loop-flag thresholds with sensitivity analysis) that independent re-derivation from public checkpoints and datasets is plausible in the meantime.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 are only partially recoverable from the text extraction; figure-level values were read from embedded annotations, captions and axis labels, not from rendered plots.
- The abstract is truncated mid-sentence in the extract, and some appendix tables (e.g., per-cell significance counts) appear as prose summaries rather than full tables.
- This is a v1 preprint; code and result tables are promised only 'on publication', so reported numbers could not be checked against released artifacts.
- All judgments rest on the paper's own reported statistics; no independent verification of headline numbers was possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.