SpeakPay: Domain-Adaptive LoRA Fine-Tuning of Whisper for Low-Resource Nepali Financial Speech Recognition
cs.CL
Quick verdict
Reports LoRA-tuned Whisper cutting Nepali financial WER 129.95%→42.58%; a credible, honest small-scale feasibility study with minor reporting ambiguities.
Summary
This paper presents SpeakPay, a voice-first digital wallet for Nepal, whose central contribution is a controlled study of domain adaptation for low-resource financial speech recognition. The author releases NepFinSpeech-403, presented as the first public Nepali financial speech dataset: 403 utterances of send, load, and balance commands (303/40/60 train/validation/test) covering 237 unique Devanagari numerals. Fine-tuning Whisper large-v2 with LoRA (rank 32, roughly 8M trainable parameters) on a single RTX 3060 cuts held-out-test WER from 129.95% (zero-shot) to 42.58%, a 67.2% relative reduction, and lifts numeral accuracy from 0.0% to 73.9%. A rule-based task evaluation raises the Transaction Success Rate from 1.67% to 33.33%, and a paired sign test finds improvement on 59 of 60 utterances (p = 3.5×10⁻¹⁸). A data-efficiency sweep indicates 100–300 utterances capture most of the gain; acoustic degradation tests show graceful decline except at 0 dB SNR. The author candidly reports limitations: small scale, possible speaker overlap across splits, a size-mismatched general-domain baseline, single-annotator transcripts, an undefined NumAcc metric, an ambiguous '403 (full)' row in the sweep table, and no usability study with visually impaired users; at 33.33% task success the system remains a feasibility demonstration rather than a deployable payment tool.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LoRA fine-tuning of Whisper large-v2 on 303 Nepali financial utterances cuts held-out WER from 129.95% to 42.58% (67.2% relative) and raises Devanagari numeral accuracy from 0.0% to 73.9%.
- Domain adaptation raises the rule-based Transaction Success Rate from 1.67% to 33.33% (~20×), showing word-level metrics understate task-level gains for slot-critical applications.
- The improvement holds on 59 of 60 individual test utterances (sign test p = 3.5×10⁻¹⁸) and across all command types.
- 100–300 domain utterances capture most of the adaptation gain, with zero-shot WER roughly halved by ~100 examples.
- Residual failures concentrate in systematic numeral confusions (zero insertion/deletion, prefix hallucination), motivating confirmation steps before executing transactions.
Strengths
First public Nepali financial speech resource
NepFinSpeech-403 (403 utterances, 237 unique Devanagari numerals, §3) is released under CC-BY 4.0 alongside MIT-licensed code and model weights, making replication and reuse genuinely possible.
Statistically grounded paired evaluation
§5.2: 59 improved / 0 reversed / 1 tied across 60 test utterances, sign test p = 3.5×10⁻¹⁸ (arithmetically consistent with the counts), plus bootstrap CIs (B = 10,000) for both models' mean per-utterance WER.
Task-level metric exposes practical impact
Actionable data-efficiency findings
Unusually candid limitation reporting
Weaknesses
Ambiguous, possibly contaminated sweep endpoint
Undefined headline metric NumAcc
Fragile plateau claim from single-run sweep
Baseline and parser confounds
Small, speaker-overlapped evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
SpeakPay — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What accuracy does SpeakPay achieve for Nepali financial speech recognition?
On its 60-utterance held-out test set, the LoRA-adapted Whisper large-v2 reaches 42.58% WER, down from 129.95% for zero-shot Whisper large-v2 (a 67.2% relative reduction), and Devanagari numeral accuracy rises from 0.0% to 73.9%.
How much data do you need to adapt Whisper to a specialized domain?
In this paper's sweep, 50 utterances already cut WER by 43% relative to zero-shot, roughly 100 utterances halve the zero-shot WER, and gains plateau around 300 examples — suggesting 100–300 utterances suffice for a prototype in similar low-resource settings.
Is SpeakPay ready for real payments?
No. The Transaction Success Rate is 33.33% and numeral accuracy 73.9%; the deployed app runs on simulated balances, and the authors state that spoken-confirmation steps are required before any real financial execution.
Where can I download the NepFinSpeech-403 dataset and code?
The dataset is on Hugging Face (birajsubedi/NepFinSpeech, CC-BY 4.0), fine-tuned model weights are on Hugging Face, and the full codebase is MIT-licensed at github.com/subedibiraj/speakpay.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction is complete — the text runs to the end of reference [19] with no mid-sentence cut; Figures 1–2 appear only as linearized label/number text (legible, but not visually inspectable).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.