SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia
Jingyi Liao, Wenyu Zhang, Zhuohan Liu, Yingxu He, Geyu Lin, Xunlong Zou, Shuo Sun, Syed Ali Redha Alsagoff, et al.
cs.CL
Quick verdict
Claims the first multitask SEA speech benchmark (97k samples, 11 languages, 9 tasks) and largely delivers; credible harness, but judge/label-validity and noisy temporal references warrant caution.
Summary
SEA-SpeechBench aggregates 97,194 samples, 99 evaluation sets and 597 hours of audio across 11 Southeast Asian languages into 9 tasks: ASR, speech translation, spoken QA, emotion/gender/age/speaker recognition, and two novel temporal tasks (Timestamped Content Query and Temporal Localization) over clips up to 180 seconds. The authors evaluate 14 open-source and commercial systems under parallel English and native-language prompts, plus a cascaded Whisper+Qwen3.6 baseline. Headline results: ASR approaches practical levels for most languages (Gemini 2.5 Flash averages 0.15% WER/CER); speech translation and emotion recognition stay weak (best ER accuracy 42.55%, Kimi-Audio); temporal-task performance degrades steeply with length (TLoc F1 falls from 26.5 at 0-30s to 5.4 at 120-180s) and most open-source models cannot ingest clips beyond 60 seconds. English prompts consistently beat native prompts, worst for Burmese, Tamil and Khmer, up to 41 points. Strengths include a human-validated judge protocol and per-dataset appendix results. Key concerns: temporal-task references derive from CTC-filtered ASR transcripts rather than human verification; the deployed emotion judge's agreement with humans weakens under SEA prompts (MCC 0.66); age/gender labels are never validated against perceived voice traits; several sources have unclear licenses; and no significance testing accompanies headline gaps.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 6 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SEA-SpeechBench is, to the authors' knowledge, the first large-scale multitask benchmark for speech understanding across 11 Southeast Asian languages (97,194 samples, 99 sets, 597 hours, 9 tasks).
- Evaluated models remain weak on temporal understanding, emotion recognition, and speech translation, with temporal-task performance degrading as audio length grows to 180 seconds.
- English prompts consistently outperform native SEA-language prompts, with gaps up to 41 points for low-resource languages such as Burmese and Tamil.
- GPT-4o's near-zero speaker-recognition score (14.49 macro-F1) is dominated by refusals (89.5%), consistent with deliberate safety-policy abstention rather than task incomprehension.
Strengths
Genuinely novel temporal-understanding dimension
TCQ and TLoc probe time-to-content and content-to-time mapping over clips up to 180s (§2.1, Table 9a); no prior audio-LLM benchmark in the related-work survey covers temporal localization, and stratified duration bins enable a clean degradation analysis.
Scale and coverage with documented provenance
97,194 samples / 99 sets / 597.49 hours over 11 official SEA languages (Table 2); Appendix A catalogs ~22 sources with per-source licenses, metadata availability and usage (Table 11), including explicit rationale for excluding YODAS2 transcripts from gold ASR references.
Judge protocol is human-validated, not assumed
Methodologically careful construction details
Transparent failure analysis
Weaknesses
Temporal-task references are machine-derived, not human-verified
Deployed ER judge has weak SEA-prompt human alignment
Generator-judge-evaluatee overlap in SQA
Paralinguistic label validity is unestablished
No uncertainty quantification; provenance and compliance gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SEA-SpeechBench (arXiv:2609.09672) claims, with an appropriate 'to our knowledge' hedge, to be the first large-scale multitask benchmark for speech understanding across Southeast Asia: 97,194 samples, 99 evaluation sets, 597.49 hours of audio, 11 official SEA languages (Table 1 includes English, reflecting Singapore/Philippine official-language status), 9 tasks in 3 categories. The genuinely novel contribution is the temporal pair — Timestamped Content Query (time→content) and Temporal Localization (content→time) — over clips up to 180 s (§2.1). Secondary claims: models stay weak on temporal understanding, emotion recognition, and speech translation; English prompting beats native-language prompting by up to 41 points; GPT-4o's SpkR failure is refusal-driven (§3.2.5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 6 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SEA-SpeechBench?
A large-scale benchmark for evaluating how well audio and speech-capable large language models understand speech in 11 Southeast Asian languages. It contains 97,194 evaluation samples across 99 evaluation sets and 597 hours of audio, covering 9 tasks: speech recognition, speech translation, spoken question answering, emotion/gender/age/speaker recognition, and two novel temporal tasks (transcribing content at a given timestamp, and locating when a given content occurs) over clips up to 3 minutes.
Which models were evaluated and what were the main results?
Fourteen systems, including MERaLiON-2, SeaLLMs-Audio, Phi-4-multimodal, Qwen2-Audio, Qwen2.5-Omni, Gemma-3n, Voxtral, Kimi-Audio, Qwen3-Omni, a Whisper+Qwen3.6 cascade, and the commercial Gemini 2.5 Flash and GPT-4o. ASR is close to usable for most languages (Gemini 2.5 Flash averages 0.15% error), but speech translation and emotion recognition stay weak (best emotion accuracy 42.55%), performance on temporal tasks collapses as clips get longer (localization F1 falls from 26.5 to 5.4), and English prompts consistently beat native-language prompts, by up to 41 points for Burmese and Tamil.
Why did GPT-4o score so badly on speaker recognition?
It mainly refused to answer: 89.5% of speaker-recognition queries were refusals, while the 10.5% it did answer were all correct. The authors attribute this to safety policies around speaker identification (a privacy-sensitive task) rather than inability, and note a contrast: Qwen2.5-Omni-7B also refuses often but makes real mistakes when it answers.
Is SEA-SpeechBench publicly available, and can its results be reproduced?
Yes — the benchmark is released at zwenyu.github.io/SEA-SpeechBench. The fixed test split carries unique IDs mapping back to source corpora; when licenses permit, processed audio is redistributed, otherwise a manifest plus reconstruction instructions is provided, and prompts/evaluation scripts are published. In principle the pipeline can be re-run, though exact numbers for proprietary APIs (Gemini, GPT-4o) depend on vendor-side model versions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (2, 3, 4, 5, 6, 9b, 14) are rendered as images in the PDF; their internal numbers were not extractable, so figure-level claims (e.g., exact PAS values per language) rely on the surrounding text.
- Non-Latin prompt examples in Appendix F show OCR garbling of Thai, Tamil and Burmese script; this affects display fidelity of examples only, not the evaluated results.
- The main text and appendices (Tables 17–25) appear complete through the end of the provided extraction; only the final table header line is cut mid-word, with no additional results content after it.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.