HaikuS2S: A Cascaded System For Responding In Verse
Devangi Sharma, Sophia Judicke, Glenda Tan, Conrad Schaumburg, Shinji Watanabe
cs.CL · cs.AI · cs.SD
Quick verdict
Cascaded ASR-LLM-TTS haiku system: 2-stage fine-tuning lifts prosody (WER .425→.324), but no stat tests, text clashes with tables, stage 1 lacks human eval.
Summary
HaikuS2S is a single-speaker cascaded spoken-dialogue system (ESPnet-SDS) that answers user queries with spoken English haiku: OWSM v4 ASR transcribes the query, a Phi-3-mini LLM writes a 5-7-5 haiku that a grapheme-to-phoneme vowel-counting loop forces to be regenerated until the form is valid, and a fine-tuned VITS TTS recites it, using a special [\n] token to elicit line-ending pauses. The TTS is fine-tuned in two stages: stage 1 on 827 poetry clips from LibriTTS-R, stage 2 on 500 haiku self-recorded by two team members at 16 kHz. On 498 test haiku recorded by two different members, stage 2 achieves the best WER (0.324 vs 0.425 baseline) and the best chroma CQT/CENS alignment, while emotion similarity stays high across systems (0.935–0.954). NISQA naturalness nonetheless drops sharply for stage 2 (MOS 3.439 vs 4.442 baseline and 4.78 stage 1), attributed by the authors to NISQA's conversational-speech bias. In a 35-listener study, stage 2 beat the baseline on all seven rated dimensions. Main limitations: significance is claimed without test names or p-values; the text's "lowest error rate" claim conflicts with Table 2 (stage 1 CER 0.081 beats stage 2's 0.099, and stage 1 WER slightly regresses); stage 1 is absent from the human evaluation; and all training/test audio comes from the authors. Model and data are promised on HuggingFace.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Two-stage TTS fine-tuning (827 general poetry clips, then 500 recorded haiku) yields the best haiku intelligibility (WER 0.324 vs baseline 0.425) and chroma alignment (CQT 9.599, CENS 7.466 vs baseline 12.169/13.710).
- A G2P vowel-counting validation loop between the LLM and TTS enforces valid 5-7-5 haiku, and a special [\n] token elicits line-ending pauses in recitation.
- Emotion similarity to gold recordings is essentially maintained across all systems (0.935 baseline, 0.946 stage 1, 0.954 stage 2; Table 1).
- Human listeners (n=35) prefer stage 2 over the baseline on all seven rated dimensions (Table 5), despite stage 2's substantially lower NISQA scores.
Strengths
Novel, well-scoped task with concrete structural enforcement
Sec. III combines a fixed 5-7-5 LLM prompt with a G2P vowel-counting regeneration loop between LLM and TTS, plus a special [\n] token in fine-tuning stages 1-2 to induce line-ending pauses — explicit mechanisms, not just aspiration; the authors correctly note no prior work fine-tunes TTS specifically on haiku.
Multi-dimensional evaluation including an IRB-approved human study
Sec. IV.2 uses VERSA for WER/CER (via owsm_large-v3 re-recognition), emotion similarity, NISQA (mos/noi/col), and three chroma DTW metrics, then adds a 35-listener study over seven dimensions (Table 5) — broader than typical 4-page system papers.
Open artifacts on an open stack
Honest reporting of an unfavorable result
Informative WER/CER divergence
Weaknesses
Text-table contradiction on the headline intelligibility claim
Statistical claims without statistical reporting
Stage 1 missing from human evaluation
Speaker/timbre confounds and internal gold references
Small, synthetic data with unevaluated content quality
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
HaikuS2S: A Cascaded System For Responding In Verse — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is HaikuS2S and how does it work?
HaikuS2S is a cascaded spoken dialogue system that answers a user's spoken query with a recited English haiku. Speech recognition (OWSM v4) transcribes the query, the Phi-3-mini language model writes a three-line 5-7-5 haiku, a vowel-counting check regenerates the poem until the syllable form is valid, and a fine-tuned VITS text-to-speech model recites it, using a special newline token to insert pauses between lines.
How does HaikuS2S enforce the 5-7-5 haiku structure?
Two mechanisms: the LLM prompt demands exactly three lines with 5, 7 and 5 syllables, and a grapheme-to-phoneme validator counts the vowels in each line after generation; if the count is wrong, the LLM is prompted to regenerate until a valid 5-7-5 poem is produced.
What results did HaikuS2S achieve?
After two-stage fine-tuning (827 poetry clips, then 500 self-recorded haiku), the final system cut word error rate from 0.425 to 0.324 versus the pretrained baseline, achieved the best pitch/harmony alignment on two of three chroma metrics, kept emotion similarity high (0.935-0.954), and was preferred by 35 human listeners over the baseline on all seven rated dimensions — though a standard speech-quality metric dropped (NISQA MOS 3.439 vs 4.442).
Is HaikuS2S code, model, or data available?
The authors state that the final system model plus the 500-haiku training set and 498-haiku test set will be publicly available on HuggingFace under the HaikuS2S account; the system is built from open ESPnet, OWSM, VITS and VERSA components.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures could not be analyzed (captions only in the extracted text); all quantitative judgments rely on appendix Tables 1-5.
- Figure numbering is inconsistent between in-text citations and captions (e.g., text cites Fig. 2 for WER and Fig. 5-7 for human evaluation while captions label them differently), so plot-level claims could not be verified.
- Human-evaluation protocol details (within- vs between-subjects, whether every listener rated both systems, stimulus matching across systems) are not stated in the text.
- Extraction appears complete (abstract, body, references, appendix tables), so confidence is only moderately reduced.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.