Automatic Lyric Transcription for Greek Songs: Scaling and Task Composition Effects in Whisper Adaptation
Maria Frangiadaki, Dimitrios Damianos, Kosmas Kritsis, Vassilis Katsouros
cs.CL · cs.SD
Quick verdict
Claims first Greek ALT benchmark via Whisper adaptation on new GAD-ALT corpus (27.2% WER); credible open study, but no significance testing or seed variance.
Summary
This paper presents the first controlled study of Automatic Lyric Transcription (ALT) for Greek, addressing the absence of any Greek singing-voice ASR benchmark. The authors curate GAD-ALT from the Greek Audio Dataset (1,000 songs) using htdemucs source separation, CTC forced alignment with a custom confidence filter, and GPT-4o-mini segment-level translations, yielding 17,458 aligned segments (19.65 hours) with song-level train/validation/test splits. They fine-tune Whisper Small, Medium, and Large-v3 under transcription-only, multitask (2:1 and 4:1 transcribe:translate), and two-stage speech-to-singing adaptation (stage 1 on Greek Common Voice with a frozen encoder). Zero-shot WERs are 92.3/65.1/53.6%; best adapted results are 33.6% (Small, 2:1 multitask), 30.1% (Medium, two-stage), and 27.2% (Large-v3, two-stage), the last establishing the headline benchmark. Multitask training helps only the smallest model, acting as regularization; the best model scores 33.4% on raw polyphonic audio versus 27.2% on isolated vocals, and all remixing augmentations degraded performance. A 200-error taxonomy highlights semantic substitutions, boundary drift, and Greek orthographic ambiguity. Main limitations: no significance testing or multi-seed variance despite tiny gaps (e.g., 30.1 vs 30.3% for Medium), no human validation of machine-generated translation targets, unreported alignment-filter retention rates, and no non-Whisper baseline such as XLS-R. Code, models, and data are stated to be openly available.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Model scaling consistently improves Greek singing transcription, with zero-shot WER falling from 92.3% (Small) to 53.6% (Large-v3) on GAD-ALT.
- Multitask transcribe-translate training acts as a beneficial regularizer primarily for smaller-capacity Whisper models (Small: 33.6% vs 36.7% transcription-only).
- Two-stage speech-to-singing adaptation achieves the best result — 27.2% WER with Whisper Large-v3 — establishing the first Greek ALT benchmark.
- Vocals-only training after source separation outperforms raw polyphonic audio (27.2% vs 33.4% WER) and all remix-based augmentations degrade performance.
Strengths
Genuine gap filled with full open release
First Greek ALT benchmark: GAD-ALT (17,458 segments, 19.65 h from 1,000 songs, §3) plus code and models on GitHub/Hugging Face under Apache 2.0/CC-BY 4.0 (footnote 1), with reproducible song-level splits (78.8/10.8/10.4%).
Clean controlled factorial design
Three Whisper scales (244M/769M/1.55B) × five regimes (zero-shot, 2:1, 4:1, transcription-only, two-stage) in Table 1, with a consistent zero-shot scaling trend (92.3 → 65.1 → 53.6% WER) and task-pure batch interleaving (§4.2).
Honest negative ablations
Language-specific error taxonomy
Weaknesses
No inferential statistics anywhere
Regularization claim rests on a single scale
Unvalidated machine-generated training targets
Missing baselines and filtering transparency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets Automatic Lyric Transcription (ALT) for Greek, a language for which the authors claim — plausibly, given the §2.3 survey — that no singing-transcription benchmark exists. Four claims are made: (1) GAD-ALT constitutes the first systematic Greek ALT benchmark; (2) model scaling consistently improves performance (zero-shot WER in Table 1: 92.3% → 65.1% → 53.6% for Small → Medium → Large-v3); (3) multitask transcription-plus-translation training acts as a regularizer mainly for smaller models (Small: 33.6% WER at a 2:1 mix vs 36.7% transcription-only, §5.2); and (4) two-stage speech-to-singing adaptation wins at scale, reaching 27.2% WER with Whisper Large-v3 (§5.3) — the abstract's headline. Claims are scoped to Greek and the Whisper family, and the paper mostly honors that scope.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the best word error rate achieved for Greek lyrics transcription?
27.2% WER, achieved by two-stage adapted Whisper Large-v3 on the GAD-ALT test set, down from 53.6% zero-shot. The authors present this as the first benchmark for Greek automatic lyric transcription.
What is the GAD-ALT dataset and is it publicly available?
GAD-ALT is a segment-level aligned Greek singing corpus derived from the 1,000-song Greek Audio Dataset: 17,458 aligned segments totaling 19.65 hours, each with a Greek transcription and an English translation, split at song level. The paper states code, models and data are on GitHub (athena-ilsp/lyrics-transcription) and Hugging Face under Apache 2.0 / CC-BY 4.0.
Does adding translation as an auxiliary task help Whisper transcribe singing?
Only for the smallest model tested: Whisper Small improved from 36.7% (transcription-only) to 33.6% WER with a 2:1 transcribe-translate training mix. For Medium and Large-v3, transcription-only or two-stage training performed better.
Does source separation matter for lyric transcription accuracy?
Yes in this study: the best model scored 27.2% WER on separated vocal stems versus 33.4% on raw polyphonic mixtures, and every remix-based augmentation tried (SNR remixing, reverberation, mixed training) degraded performance.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 appear only as images with captions in the extraction; their plotted data points are not recoverable from the text, so figure-based trends were assessed from the surrounding prose alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.