DiaWhisper-DPO: Role-Attributed Transcription of Clinical Interviews via Failure-Mined Preference Optimization
Weiming Li, Ana Catarina Fidalgo Barata, Miguel Constante, João Miguel Sanches
cs.CL
Quick verdict
Whisper + failure-mined DPO for clinician/patient-tagged transcription; impressive 0.973 role accuracy but on synthetic DAIC audio, vs cascaded-only baselines.
Summary
The paper proposes DiaWhisper, an end-to-end model that fine-tunes Whisper-large-v3 with LoRA and an auxiliary frame-level role-classification head to jointly transcribe two-party clinical interviews and attribute each utterance to clinician or patient. A second stage, DiaWhisper-DPO, mines genuine decoding failures (8-token repetition loops) from the training set as DPO rejected completions, with ground truth as chosen completions, requiring no human preference annotation. Because DAIC-WOZ distributes only participant audio, the authors re-synthesize both conversation sides with XTTS and quantify the resulting acoustic gap (patient-side WER 0.136 real vs 0.106 synthetic); cross-lingual validation uses voice-converted real Chinese interviews (PDCH-HAMD, 100 sessions). On 29 DAIC-WOZ test sessions, DiaWhisper-DPO reaches 0.973 role accuracy and 0.119 DER versus 0.431 for the strongest cascaded baseline (CAM++), with seed variability falling from σ=0.205 to 0.002; on 26 PDCH-HAMD sessions it reaches 0.757 role accuracy and 0.313 DER, improving all 78 session-seed pairs. Ablations show the DPO signal outperforms extra SFT (0.684) and artificial negatives (0.910, with 18× seed variance), and the retry mechanism alone does not explain the gains. Main limitations: English evaluation rests entirely on synthetic two-party audio, Chinese residual error is dominated by wrong-role attribution (25.8%), and only cascaded baselines are compared — no end-to-end joint systems.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Failure-mined DPO — genuine decoding failures as rejected completions, ground truth as chosen — improves role-attributed transcription more strongly and stably than extra SFT or artificial negatives, at no annotation cost.
- On 29 DAIC-WOZ test sessions, DiaWhisper-DPO achieves 0.973 role accuracy and 0.119 DER, a 72% DER reduction relative to the strongest cascaded baseline (CAM++, 0.431).
- DPO cuts seed variation in role accuracy from σ=0.205 to 0.002 on DAIC-WOZ and from σ=0.052 to 0.005 on PDCH-HAMD.
- Retrained independently on voice-converted Chinese PDCH-HAMD, the method reaches 0.757 role accuracy and improves all 78 matched session-seed pairs, supporting cross-lingual applicability.
- Gains are attributable to the preference signal rather than retry recovery or extra ground-truth exposure: retry ablation barely affects DiaWhisper-DPO (0.973→0.971) and SFT substitution reaches only 0.684.
Strengths
Annotation-free preference signal mined from genuine failures
Stage 2 (Sec 2.3) mines ~500 training windows per seed that trigger the 8-token repetition criterion and pairs the model's own failed decoding (rejected) with ground truth (chosen), needing no preference annotation. Ablations (Table 2) show this beats extra SFT (0.973 vs 0.684 role accuracy on DAIC-WOZ) and artificial negatives (0.910, with 18× the seed SD of 0.037 vs 0.002).
Unusually rigorous, multi-angle evaluation protocol
Session-bootstrap 95% CIs with 2000 resamples pooling 3 seeds (Sec 3.1), coverage-conditioned per-role AER (Eq 7), oracle role-mapping checks separating attribution from coverage (baselines shift only 0–7 points; DiaWhisper-DPO still +21.5/+17.1 ahead), and matched-turn accuracy deltas with non-overlapping CIs (95.4→98.3% DAIC; 62.3→75.9% PDCH).
Honest quantification of the synthetic-data caveat
Ablations isolate the active ingredient
Cross-lingual validation on real interviews
Weaknesses
Headline English results are on fully synthetic audio
No empirical comparison with cited end-to-end joint systems
Transcription advantage does not hold on Chinese; residual errors large
Seed-stability claim rests on 3 seeds with no significance testing
Baseline tuning opacity, no code release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets role-attributed transcription of two-party clinical interviews: joint ASR plus attribution of each utterance to clinician or patient, motivated by depression screening where speaker identity conditions symptom interpretation (Sec 1, refs [1–3]). Two corpora play complementary roles: DAIC-WOZ (132/28/29 sessions), re-synthesized into two-party audio with XTTS because only participant audio is distributed, and PDCH-HAMD (100 voice-converted real Chinese consultations, 73 patients, patient-wise splits). Central claims: (i) failure-mined DPO beats extra SFT and artificial negatives at no annotation cost; (ii) DiaWhisper-DPO reaches 0.973 role accuracy and 0.119 DER on DAIC-WOZ, 72% below the best cascaded baseline; (iii) seed variance falls from σ=.205 to .002; (iv) the approach transfers to Chinese (0.757 role accuracy, all 78 session-seed pairs improved).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DiaWhisper-DPO?
A fine-tuned Whisper-large-v3 model that transcribes two-party clinical interviews while tagging each utterance as clinician or patient. A second training stage applies direct preference optimization (DPO) in which the model's own real decoding failures (8-token repetition loops) serve as the negative examples and the ground-truth transcript as the positive example, so no human preference annotation is needed.
How accurate is it?
On 29 DAIC-WOZ test sessions the paper reports 0.973 role accuracy and 0.119 diarization error rate (DER), roughly a quarter of the strongest cascaded baseline's 0.431. On 26 real (voice-converted) Chinese interview sessions it reaches 0.757 role accuracy and 0.313 DER. Important caveat: the English evaluation uses speech re-synthesized with text-to-speech, not real two-party recordings.
Why do the authors synthesize the DAIC-WOZ audio?
DAIC-WOZ publicly distributes only the participant's audio, so there is no real clinician track to train or evaluate on. The authors rebuild both conversation sides from ground-truth transcripts using XTTS text-to-speech with VCTK voices, preserving exact role boundaries and timestamps, and separately measure the acoustic gap this introduces (real vs synthetic patient WER 0.136 vs 0.106).
Does it require human-labeled preference data?
No. The DPO stage mines about 500 training windows per seed where decoding hit the 8-token repetition failure criterion, uses the model's own failed output as the rejected completion and the ground-truth transcript as the chosen completion, and trains for one epoch — no additional annotation is involved.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 are referenced but only their captions/prose descriptions are extractable; Fig 2 readings (e.g., 25.8% wrong-role) come from the text, not the plot.
- Per-system oracle role-mapping values are explicitly omitted by the authors ('for space'), so that check cannot be independently verified from the text.
- arXiv extraction may drop minor formatting; all numbers cited here were cross-checked against Tables 1–2 and the prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.