Benchmarking and Domain Adaptation of Automatic Speech Recognition (ASR) for Adolescent Health Communication in Ghanaian Languages
Stephen E. Moore, Akwasi Asare, Mich-Seth Owusu, Paul Azunre, Joel Budu, Lawrence A. Adu-Gyamfi
cs.CL · cs.AI
Quick verdict
Three-stage ASR benchmark, fine-tune and deployment for Ghanaian languages; valuable open pipeline, but evaluation is small-n with no statistical testing.
Summary
The paper reports a three-stage study of ASR for adolescent sexual and reproductive health (ASRH) communication in Twi, Dagbani, and Ewe. Stage one benchmarks five systems — three language-specific Wav2Vec2-BERT models and two zero-shot multimodal LLMs (Gemma 3n, Gemma 4) — on Bible-corpus and Youth-ASRH audio (50 samples per condition), using jiwer-based CER/WER. Fine-tuned Wav2Vec2 models lead on CER (17.0–25.2%); Gemma 3n hallucinates (WER up to 344%); Gemma 4 emerges as a stable zero-shot baseline (WER 92.5–98.8%). Stage two pivots from an infeasible Gemma 4 fine-tune to Qwen3-ASR-0.6B, fine-tuned on ~90k out-of-domain Bible samples and evaluated on held-out human-collected in-domain audio: WER falls for all languages, most for Ewe (109.3% to 64.8%; CER 65.1% to 24.9%). Stage three reports KasaHealth, a voice-first ASRH app running the production Khaya pipeline (not the fine-tuned model), tested by 50 community respondents — 100% chat approval (26 interactions; only 1 from Ewe), 72% translation Good/Excellent, 92% would recommend — plus Senti-Check, a blind emotion-proxy harness where the fine-tune doubles base accuracy (16/120 vs 8/120). Main limitations: no significance testing or confidence intervals, small evaluation sets, no comparison of the fine-tuned model against Wav2Vec2/Gemma 4 on the same data, and missing Whisper/MMS baselines. All data, weights, and tools are open-source.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Gemma 4 is a newly competitive zero-shot ASR baseline for Twi, Dagbani, and Ewe (WER 92.5–98.8% across six conditions), evaluated here for the first time.
- Fine-tuning Qwen3-ASR-0.6B on ~90k out-of-domain Bible samples reduces WER on held-out in-domain audio for all three languages, most dramatically for Ewe (109.3% to 64.8%).
- For these languages, the binding constraint is validated in-domain data, not model capability or computation.
- KasaHealth demonstrates community acceptance of voice-first ASRH delivery (100% chat approval, 72% Good-or-Excellent translation, 92% would recommend).
- Senti-Check indicates the fine-tune doubles downstream semantic accuracy in a blind emotion-classification proxy (16/120 vs 8/120).
Strengths
Complete research-to-deployment arc
Unusually for low-resource NLP, the paper closes the loop: benchmark (Section 4.1), adaptation (4.2), and a real user study with 50 community respondents on a deployed app (4.3.1), rather than stopping at offline metrics.
Strict train/eval domain separation
Training used only out-of-domain Bible audio (~90k samples) while evaluation used held-out human-recorded in-domain audio (Section 3.3, Table 1); the authors honestly frame Table 3 results as 'a conservative lower bound', avoiding a common contamination trap.
Documented negative result with practical value
Full open-source release
Transparent component-level UAT reporting
Weaknesses
No inferential statistics anywhere
Headline metrics rest on tiny, unbalanced denominators
No same-set comparison for the fine-tuned model
Missing standard baselines
Underspecified evaluation sets and WER>100% reporting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper spans benchmarking, domain adaptation, and deployment for ASR in Twi, Dagbani, and Ewe, targeting adolescent sexual and reproductive health (ASRH) communication. Three headline claims: (1) Gemma 4 is a newly competitive zero-shot ASR baseline (WER 92.5–98.8% across six conditions, Section 4.1); (2) fine-tuning Qwen3-ASR-0.6B on ~90k out-of-domain Bible samples reduces WER on held-out human in-domain audio for all three languages — Twi 106.6→95.1%, Dagbani 99.1→88.1%, Ewe 109.3→64.8% (Table 3); (3) validated in-domain data, not models or compute, is the binding constraint. A deployment claim is added via KasaHealth (100% chat approval, 72% translation Good/Excellent, 92% would-recommend). Claims are clearly stated and mostly well-scoped, though the abstract's '100% chat-approval' is a headline built on a small, unbalanced denominator (26 interactions, one from Ewe).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which languages and ASR models were compared in this study?
Three Ghanaian languages — Twi, Dagbani, and Ewe. Five systems were benchmarked: three language-specific Wav2Vec2-BERT models and two zero-shot multimodal LLMs (Gemma 3n and Gemma 4). A compact Qwen3-ASR-0.6B was later fine-tuned for domain adaptation.
What did fine-tuning the ASR model achieve?
The model was trained on about 90,000 out-of-domain Bible speech samples and evaluated on strictly held-out human-recorded in-domain audio. Word error rate fell for all three languages, most for Ewe: from 109.3% to 64.8% (character error rate from 65.1% to 24.9%). Twi and Dagbani improved by about 11 percentage points each.
Does the KasaHealth app use the paper's fine-tuned model?
No. KasaHealth runs Ghana NLP's production Khaya pipeline, not the fine-tuned Qwen3-ASR model. The fine-tuned model was validated separately via Senti-Check, a blind emotion-labelling proxy task where it scored 16/120 versus the base model's 8/120.
Are the datasets, models, and code from this paper publicly available?
Yes. The paper states a full open-source release: Hugging Face datasets (Ghana Bible corpus ~90k samples, UNICEF Youth ASRH ASR audio), fine-tuned model weights, and GitHub tools (offline recorder, Senti-Check harness), plus a live KasaHealth web application.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–7 are referenced but not analyzable from the extracted text; all numeric claims cited here come from tables and prose.
- Table 2's extraction is garbled (multi-column layout flattened); values were reconstructed carefully but a misparse remains possible.
- Evaluation-set sizes for Table 3 (utterances, speakers) are not stated anywhere in the text, limiting verification of the central adaptation result.
- The 'Gemma 4' model is cited only via a Google docs page (ref 26), not a technical report; the model version could not be verified from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.