MedWER: A Reproducible, Model-Free Evaluation Protocol for Medical Speech Recognition
cs.CL
Quick verdict
MedWER scores medical ASR against a fixed 19,373-term list instead of NER; a careful, honestly-limited protocol whose three validations are not reproducible.
Summary
The paper argues that overall word error rate hides clinically critical mistakes on drug and diagnosis names, while entity-weighted metrics that rely on evaluation-time NER models make the metric's denominator a versioned black box. MedWER replaces the NER dependency with a fixed, license-clean term list of 19,373 drug, diagnosis, symptom, and injury-mechanism entries projected from Health Canada's DPD/CCDD and ICD-10-CM. A pinned Whisper text normalizer, checked against golden fixtures on every commit, feeds a greedy longest-match filter that scores multi-word terms as atomic units, yielding a deterministic medical-WER. Coverage is validated against BC PharmaCare data excluded from construction (78.6% generic token recall after cleaning); matching is calibrated against Eka's human entity spans (95.7% token precision, 20.2% recall against full annotation breadth); an scispaCy NER denominator cross-check preserves system orderings. Baselines for Moonshine base (fp32/int4), Whisper base.en, and MedASR on Eka-test (543 utterances) and PriMock57 (3,405) come with bootstrap 95% confidence intervals. The metric surfaces effects overall WER hides, e.g., MedASR reaches 18.6% medical-WER on Eka-test versus 30.6% for the best general-purpose model despite 42.8% overall WER on PriMock57. Limitations include narrow vocabulary recall, 49 common-English entries counted everywhere, and three validation studies that cannot be reproduced from the released artifact; the author's company sells a dictation product evaluated with the tool.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- NER-dependent medical WER metrics have versioned black-box denominators, and a fixed license-clean term list with a pinned normalizer makes medical-WER deterministic and reproducible.
- The 19,373-entry list reaches 78.6% generic drug token recall on an independent BC PharmaCare holdout (826/1,051 after cleaning) and 95.7% token precision against human entity spans.
- Replacing an scispaCy NER denominator with the fixed list supports the same or fewer orderings but reverses no supported system ordering on either benchmark (§5.2).
- Entity-restricted scoring separates domain competence from register mismatch: MedASR leads Eka-test medical-WER at 18.6% while its overall WER collapses to 42.8% on PriMock57.
- Filter-then-align is a protocol-defining choice whose ablation (align-then-count, 0.8–1.7 points higher across all eight Table 3 cells) shifts levels but preserves every supported ordering.
Strengths
Reproducibility engineering is exemplary
The pinned normalizer is asserted against 22 normalization, 14 WER, and 9 medical-WER golden fixtures on every commit (§4.1); every term-list entry is a fixed point of the normalizer (§3.3); version 1.0.0 pins the denominator (§9); and eval/ ships the 543 Eka-test utterance ids plus a rebuild script that reproduces the Eka columns of Table 3 exactly (§6).
Unusually honest validation of a heuristic
Coverage is measured against a deliberately held-out BC PharmaCare file (72.6% raw, 78.6% after reference-artifact cleaning, §5); span calibration reports the flattering 95.7% token precision alongside the unflattering 20.2% recall (§5.1); and the deviation of greedy from optimal matching is disclosed as unmeasured (§8).
Design decisions are backed by ablations, not assertion
License-clean provenance reasoned end to end
Findings that matter operationally
Weaknesses
Validation studies are not reproducible from the release
Narrow denominator by design
Membership-vs-use contamination is unquantified
Moderate drug coverage, and missing terms fail silently
Single author, commercial interest, thin benchmark spread
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MedWER (arXiv 2609.05728) is a method paper introducing an evaluation protocol and open-source tool for medical ASR whose entity-restricted word error rate has a fixed, versioned denominator: a 19,373-entry term list of drugs, diagnoses, symptoms, and injury mechanisms projected from Health Canada's DPD/CCDD and ICD-10-CM (§3), replacing the evaluation-time NER models or APIs used by prior entity-weighted metrics (§2). The central claims are: (1) NER-based medical WER couples its denominator to a versioned black box, and a fixed license-clean list removes that; (2) the list reaches 78.6% generic drug token recall against a BC PharmaCare holdout deliberately excluded from construction (826/1,051 after cleaning, §5) and 95.7% token precision against Eka's human entity spans (§5.1); (3) swapping a scispaCy NER denominator for the fixed list reverses no supported system ordering on two benchmarks (§5.2, Table 2); (4) entity-restricted scoring separates domain competence from register mismatch. Contributions are explicitly operational, and claims are scoped to a declared English/Canadian vocabulary.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MedWER and how does it differ from ordinary WER?
MedWER is a medical-domain variant of word error rate for ASR evaluation. Ordinary WER counts every word equally; MedWER filters both the reference and the hypothesis to content matching a fixed, versioned list of 19,373 medical terms (drugs, diagnoses, symptoms, injury mechanisms) and computes WER only on that content, scoring multi-word terms like 'insulin glargine' as single atomic units. The term list replaces the NER model that other medical WER variants run at evaluation time, so the metric is deterministic and reproducible.
Is the MedWER code and term list publicly available?
Yes. The scorer is MIT-licensed and installable via 'pip install medwer', with the term list shipped inside the package. Code, term list, and golden fixtures are on GitHub (Nordis-Tech/medwer) and the v1.0.0 source is archived on Zenodo (DOI 10.5281/zenodo.22103865). The paper's three validation studies are not reproducible from this release because their inputs are not redistributable.
How well does the fixed term list cover medical vocabulary?
Against an independent British Columbia drug-benefit file excluded from its construction, the list covers 78.6% of generic drug ingredient tokens after cleaning (72.6% raw) and 61.9% of brand-name tokens. Against the full breadth of human-annotated medical entities in the Eka corpus it reaches 20.2% recall by design: the list deliberately covers drugs, diagnoses, symptoms, and injury mechanisms in a Canadian scope, not all medical language, and laboratory and organism vocabularies are absent.
Which ASR models were evaluated and what were the results?
Moonshine base (fp32 and 4-bit quantized), Whisper base.en, and MedASR, all decoded greedily on two public benchmarks. On Eka-test (543 utterances) MedASR achieved 18.6% medical-WER versus 30.6% for the best general-purpose model (Moonshine fp32); on PriMock57 (3,405 utterances) MedASR's overall WER collapsed to 42.8% while its medical-WER (24.5%) stayed competitive.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.