SEABED: SouthEast Asian Benchmark for Evaluating Audio Reasoning
Harshit Rajgarhia, Asif Shaik, Rachuri Lokesh, Sushanta Kumar Pani, Abhishek Mukherji, Abhishek Mukherji
cs.SD · cs.AI
Quick verdict
SEABED: solid reasoning-first SEA audio benchmark (best model 50.3%); rigorous ablations, but LLM-authored golds and no human validation temper conclusions.
Summary
SEABED is a benchmark of 5,404 audio question-answering pairs for evaluating reasoning over Southeast Asian speech, built entirely from ten openly licensed corpora in Indonesian, Thai, and Malay. Its six tasks — dialect/language identification, prosodic ambiguity resolution, dialectal speech comprehension, long-form audio reasoning, speech emotion recognition, and speech-affective interpretation — each isolate a phenomenon that transcripts alone cannot resolve, with ten-option multiple-choice and open-ended formats split evenly within every task. Evaluating six audio LLMs, Gemini 3.5 Flash leads at 50.3% weighted accuracy, while region-specific models trail (MERaLiON-3-10B 29.4%, SeaLLMs-Audio-7B 17.5%). A two-axis audit of stated reasoning shows accuracy and grounding diverge: pooled across models, 65.8% of correct dialect-identification answers are lucky guesses, and errors are comprehension-dominated (56.9%). Five ablations (transcript substitution, question-language translation, option cardinality, tonal confounding, answer format) support design validity: transcripts cost the best model 9.2 points over four content tasks, and shrinking options to four inflates weak models most. Main limitations: no systematic human validation or human baseline; QA for two tasks (1,372 items) generated by Gemini 3.1 Pro while three evaluated models are Gemini-family; open-ended scoring by a single LLM judge without human agreement statistics; no confidence intervals on headline results; and heavily imbalanced emotion labels (Fear: one item).
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SEABED is the first reasoning-first, audio-first benchmark for Southeast Asian speech: six tasks, 5,404 QA pairs, built entirely from openly licensed corpora with no new or synthesized audio.
- The benchmark is unsolved: the best of six evaluated audio LLMs, Gemini 3.5 Flash, reaches only 50.3% weighted accuracy, while region-specific models trail far behind (MERaLiON-3-10B 29.4%, SeaLLMs-Audio-7B 17.5%).
- Answer accuracy and reasoning grounding diverge: up to a quarter of audited correct answers have ungrounded stated reasoning, and 65.8% of pooled correct dialect-identification answers are lucky guesses.
- Failures are comprehension-dominated (56.9%) rather than perceptual (25.2%), except on dialectal speech comprehension where perception errors dominate (62.2% of Gemini's DSC errors).
- Benchmark scores are not design artifacts: audio beats transcripts (+9.2 points for the best model over four content tasks), rankings survive the answer format, four-option formats inflate weak models most, and neither question language nor tonality biases results.
Strengths
Task design makes audio genuinely load-bearing
The AMBIGUOUS task uses structurally ambiguous Indonesian sentences (STRUCT_AMB_IND) whose intended reading was fixed at recording time, so a text-only system is provably at chance; Table 3 confirms this with a +39.6-point audio-over-transcript delta for Gemini 3.5 Flash, and DIALECT stimuli control content so identification must come from phonology alone (§3.3.1–3.3.2).
Unusually broad robustness/ablation suite
Five ablations (§4.4–4.8): transcript substitution (+9.2 for the best model over four content tasks), question-language translation (−5.5 to +0.7, showing the English default does not distort scores), option cardinality (10→4 options inflates weak models up to +18.5 vs +9.9 for the strongest), tonal confounding (label-controlled, no penalty), and answer format (identical rank order across MCQ/open-ended).
Honest, specific self-criticism
Cross-family anti-self-grading protocol
Construction transparency and anti-shortcut engineering
Weaknesses
No human validation of gold labels anywhere
Statistical reporting is thin
Gemini-family circularity is acknowledged but unquantified
Degenerate emotion-label distribution
Internal inconsistency suggests prompt reuse
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SEABED (arXiv:2609.22586, Centific AI Research) targets a real gap: audio-language-model evaluation for Southeast Asian speech tests recognition, not reasoning. The positioning is precise — English reasoning suites (MMAU, MMAR, MMSU, MMAU-Pro) carry a documented validity flaw (models answer ~49.8% of MMAU from text priors with audio silenced, per He et al. 2026), while SEA suites (SeaBench-Audio, SEA-SpeechBench) inherit recognition/translation taxonomies. Four contributions are claimed: a six-task, 5,404-pair audio-first benchmark; a two-axis listen-and-reason audit; construction purely from open corpora; and a quantified accuracy-grounding gap. The framing caveat: the paper brands itself a *reasoning* benchmark, yet its own audit attributes only 3.8% of errors to reasoning failures (comprehension: 56.9%) — the tasks mostly measure audio-grounded comprehension, which is fine, but the branding overreaches slightly.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SEABED benchmark?
SEABED (SouthEast Asian Benchmark for Evaluating Audio Reasoning) is a set of 5,404 audio question-answering pairs testing whether audio-language models can reason over Southeast Asian speech — Indonesian, Thai, and Malay — rather than merely transcribe it. It has six tasks (dialect identification, prosodic ambiguity resolution, dialectal comprehension, long-form audio reasoning, emotion recognition, and affective interpretation) built entirely from openly licensed corpora.
Which model performs best on SEABED?
Gemini 3.5 Flash leads with 50.3% weighted average accuracy. Gemini 3.6 Flash follows at 47.4%, Gemini 2.5 Pro at 41.6%, and GPT-Audio 1.5 at 32.0%. The region-specific models trail badly: MERaLiON-3-10B at 29.4% and SeaLLMs-Audio-7B at 17.5%.
Is SEABED data publicly available?
A sample of the benchmark is on Hugging Face (CentificAIResearch/SEABED), and the paper states item-level audit drill-down files are released with the benchmark. However, the paper itself notes the benchmark lacks systematic human validation and a human performance baseline, which the authors plan as future work.
Why do existing audio benchmarks fail to test real listening, and how does SEABED differ?
Prior work cited in the paper found models answer roughly half of the English MMAU benchmark correctly with the audio silenced, using text priors alone, and existing Southeast Asian suites mostly test recognition, translation, and classification. SEABED's tasks are designed so the answer cannot come from a transcript: distractors are anchored to audible confusability, and a transcript-substitution ablation shows removing the audio costs the best model 9.2 accuracy points.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Parts of §4.3 and Appendix Tables 9/12/13 were truncated in the extracted text; audit and ablation numbers are cited only from surviving passages.
- Figures 1-5 are not analyzable from text (only captions and example content were extracted), including the Wilson-interval plot in Figure 5.
- The paper says 'a sample of the benchmark data is available' on Hugging Face; whether the full 5,404-item benchmark and evaluation code are publicly released cannot be confirmed from the text.
- Exact sample sizes and stratification details for the 10% audit and ablation samples were not visible in the extraction, limiting precision of the statistical assessment.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.