Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projection
Maryam Abbasihafshejani, Murtuza Jadliwala
cs.AI
Quick verdict
Claims training-free decoder projection cuts Whisper non-speech hallucination 88-92% (HR 31.3→2.4-3.7%); numbers consistent, but no significance tests or code.
Summary
The paper proposes a training-free, inference-time mitigation for Whisper's tendency to emit fluent transcripts on non-speech audio. From ESC-50 folds 1–3 calibration data, the authors collect decoder hidden states for non-speech clips that trigger hallucinations versus those correctly rejected, take activation differences, and compute an SVD whose top-r right singular vectors define a rank-r projection basis at one decoder layer. At inference, hidden states are projected away from this subspace (strength α); a gated variant applies the projection only when Whisper's baseline no-speech probability exceeds γ, requiring a second decode pass. Using Whisper small, medium, and large-v3 with per-scale parameters (e.g., large-v3: ℓ=28, r=4, α=1.0, γ=0.05), always-on projection lowers the average hallucination rate over ESC-50 folds 4–5, UrbanSound8K, and filtered FSD50K from 31.31% to 2.44%, and gated projection to 3.74% (92.21% and 88.05% relative). The gated variant beats BoH phrase filtering (44.09% average HR) and slightly beats WhisperX VAD on HR (4.09% vs 5.24%), but with worse LibriSpeech WER (6.17/6.57 vs 2.67/4.74 on test-clean/other) and nonzero false rejections (up to 9.97%). Main limitations: no significance testing or variance estimates, unspecified activation-pairing details, no code-release statement, and real speech-recognition costs.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A rank-1–4 subspace, obtained by SVD of decoder activation differences between hallucinating and correctly rejected non-speech calibration clips at a single mid/late layer, captures Whisper's non-speech hallucination behavior and transfers unchanged across datasets.
- Always-on projection lowers average hallucination rate across small/medium/large-v3 on ESC-50 folds 4–5, UrbanSound8K, and filtered FSD50K from 31.31% to 2.44% (92.21% relative), with gated projection reaching 3.74% (88.05%).
- Gated projection confines the speech-recognition cost to absolute WER increases of 0.33–4.39 percentage points and false-rejection rates of 0.41–9.97% on LibriSpeech, exposing a controllable HR–WER–FRR trade-off.
- The method outperforms training-free baselines (BoH phrase filtering, no-speech threshold tuning) and exceeds WhisperX VAD on hallucination rate (4.09% vs 5.24% average) without an external VAD, though at a worse WER/FRR operating point.
- False rejections need not be permanent losses: uncertain segments can be flagged for re-transcription, verification, or human review in deployment.
Strengths
Internally consistent aggregate numbers
The abstract's headline averages re-derive exactly from Table 1 across three model scales × three datasets (small 19.06, medium 27.66, large-v3 47.23 → 31.32% original; 2.45% always-on; 3.74% gated), and the relative reductions (92.21%/88.05%) check out — unusually verifiable for an ML paper.
Honest, explicit trade-off reporting
WER and FRR get dedicated tables (Tables 2–3); gating is motivated by always-on's speech damage (large-v3 test-clean WER 12.95% vs 6.17%; FRR 10.50% vs 2.86%), and the abstract and §5.3 state the costs plainly.
Genuine cross-dataset generalization with held-out calibration
Broad baseline and ablation coverage
Weaknesses
No statistical rigor
Underspecified method details and no code
Strongest baseline wins on speech preservation
Table anomalies and config inconsistencies
No compute/latency analysis of two-pass gating
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What counts as a Whisper “hallucination” in this paper?
Here, a hallucination is any non-empty transcript Whisper produces for audio containing no speech at all (e.g., silence, roosters, car horns, jackhammers), since the correct transcript is empty. It is measured as the hallucination rate (HR): the fraction of non-speech clips yielding text after Whisper's default no-speech filtering (threshold 0.6).
Does the method require retraining or fine-tuning Whisper?
No. The method estimates a low-rank projection basis once from decoder activations on non-speech calibration audio (ESC-50 folds 1–3) and applies it at inference via a forward hook on one decoder layer. No Whisper parameters are updated, and no external VAD or phrase list is needed.
How much does this hurt normal speech transcription?
With the recommended gated variant, LibriSpeech word error rate rises by 0.33–4.39 percentage points across model scales, and 0.41–9.97% of speech clips are falsely rejected as no-speech. For comparison, the WhisperX VAD baseline achieves a similar hallucination rate (5.24% vs 4.09% average) with lower WER (2.67/4.74) and 0% false rejections, but requires an external VAD model.
Is code available for this paper?
The paper does not mention any public code, calibration bases, or checkpoint release. It does specify per-model hyperparameters (e.g., large-v3: layer 28, rank 4, α=1.0, γ=0.05) and uses fully public datasets and the official OpenAI Whisper implementation, so reimplementation is possible in principle.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- In-text figure grids (Figures 1, 2, 5–8) were extracted as flattened number runs, so per-cell figure values were used sparingly; all tables quoted in this review extracted cleanly. No truncation observed — the text ends cleanly at Table 8.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.