MuLA-Bench: A Multilingual Long-Form Audio Understanding Benchmark via Multi-Tier Auditing
Zeyu Yang, Xinyu Zhang, Zibo Bi, Pei Zhang, Xize Cheng, Jin Xu, Baosong Yang, Satoshi Nakamura
cs.SD · cs.CL
Quick verdict
记录详实的 16 种语言长音频基准测试；条件难度的研究结果可信，但基于 LLM 构建的黄金标准（gold）和上下文限制影响了模型间的对比。
Summary
MuLA-Bench 是一个多语言长音频基准：在 1,769 个野外录音（总计 1,377.9 小时；中位数 36.8 分钟）上提出 5,038 个开放式问题，涵盖 16 种语言和 8 个领域。语义轨道在每个语言×领域单元格中固定了恰好 30 个问题（8 个事实性、8 个推理、7 个时间性、7 个长距离），从而支持加性残差分析；包含 1,198 个问题的声学轨道则保留了自然发生的非语音事件，而非注入的声音。构建以证据为先——使用 GPT-5.6 Sol 进行挖掘、规划和验证；TimeAudio 加上 Gemini 3.1 Pro 进行两轮声学验证——并结合了捷径消融实验以及每位语言一名付费专家的最终审查（超过 90% 的候选项目未经修改即被接受）。在评估的十个模型中，Gemini 3.8 Flash 以 73.40% 的总体得分领先，而 MOSS-Audio-8B 为 13.48%；开源权重模型受限于上下文限制，仅能处理前缀输入。八模型池的诊断显示，语言排名在领域和任务之间存在差异，声学减去语义的差距取决于具体操作（Gemini 3.1 Pro 的语义事实性得分为 89.75%，而声学事实性仅为 17.07%，而 MOSS 的声学推理得分高于语义推理），且模型通常能恢复事件但错误报告时间（±3 秒容差）。对于 120 分钟以外的证据，长距离准确率保持在 64.0%–69.7% 之间。主要局限：由 LLM 生成黄金标准并由 LLM 进行判断（117 个样本的审计中一致性为 97.4%），每位语言仅有一名专家，以及描述性而非因果性的对比。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Language difficulty on long-form audio is conditional: rankings change across domains and tasks rather than forming a single intrinsic ordering (§6.2, Fig. 4).
- Acoustic-minus-semantic performance gaps depend on the requested operation — uniformly negative for factual items but sign-reversed for reasoning in six of ten models (§6.3, Fig. 5).
- Systems can identify the correct event yet fail physical clock alignment, with ordering and time-point accuracy dissociating across models (§6.5, Table 3).
- Long-range retrieval is comparatively robust (64.0–69.7% accuracy for evidence up to 120 minutes out), while precise clock grounding and natural acoustic factual grounding remain fragile (§6.4–6.5).
- A single long-context aggregate score conceals these conditional failure patterns (Abstract, §7).
Strengths
Controlled Language×Domain allocation enables genuinely conditional analysis
Exactly 30 semantic questions per cell (Eq. 2, §3) with fixed task mix supports the additive-residual decomposition (Eq. 1), paired-bootstrap CIs (Fig. 4a bold cells) and a cross-model SD map (Fig. A1: Vietnamese–Product SD 14 pp), moving multilingual claims beyond single aggregate rankings.
Natural acoustic evidence with tiered verification instead of injection or translation
TimeAudio proposals over full recordings, two-pass Gemini 3.1 Pro checks (blind listen then keep/modify/delete, B.4), localized audibility rechecks, and the task-aware transcript ablation (Eq. 5) target shortcut leakage while preserving wild recordings; 56 events and 8 sources were ultimately rejected (B.10).
Statistically disciplined reporting
Operation-decomposed temporal analysis with deterministic scoring
Weaknesses
Single-family LLM builds and judges the benchmark; no inter-annotator agreement
Gemini 3.1 Pro helped build the acoustic evidence it is evaluated on
Headline ranking conflates ability with context truncation
Small, uneven acoustic slices carry under-qualified weight
Minor internal inconsistencies and decoding control gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MuLA-Bench (arXiv 2609.23416) is a benchmark paper for audio-language models on long-form, multilingual, in-the-wild audio. The dataset: 5,038 open-ended questions over 1,769 recordings (1,377.9 h; median 36.8 min, 3-h cap), spanning 16 languages × 8 domains. Two tracks: a semantic track with exactly 30 questions per Language×Domain cell (8 factual / 8 reasoning / 7 temporal / 7 long-range; Eq. 2) and an acoustic track of 1,198 questions over naturally occurring non-speech events. Central claims: (i) language difficulty is conditional on domain and task, not a single ranking; (ii) acoustic–semantic gaps depend on the requested operation; (iii) event identification dissociates from clock alignment; (iv) a single long-context score hides these patterns. All four are supported within the paper's explicitly descriptive framing.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MuLA-Bench and how large is it?
MuLA-Bench is a benchmark for evaluating audio-language models on long, real-world recordings. It contains 5,038 open-ended questions over 1,769 in-the-wild recordings totaling 1,377.9 hours (median 36.8 minutes), spanning 16 languages and 8 content domains. A semantic track fixes 30 questions in each of the 128 language–domain cells; a 1,198-question acoustic track uses naturally occurring non-speech sounds like laughter, applause and music.
Which models perform best on MuLA-Bench?
Hosted models dominate: Gemini 3.8 Flash scores 73.40% overall and Gemini 3.7 Flash 73.12%. Open-weight systems score far lower — Qwen3-Omni-30B 24.91%, Audio Flamingo Next 13.60%, MOSS-Audio-8B 13.48% — partly because they can only ingest a prefix of the audio within their context limits, so their scores measure the complete system rather than pure ability.
What is the paper's key finding?
Long-audio performance is conditional, not a single number: language rankings change depending on domain and task; acoustic-vs-semantic gaps flip sign by operation type (e.g., Gemini 3.1 Pro scores 89.75% on semantic factual questions but 17.07% on acoustic factual ones, while MOSS does better on acoustic than semantic reasoning); and models often identify an event correctly but report the wrong clock time under a ±3 second tolerance.
Is MuLA-Bench publicly available?
Yes. The paper points to github.com/QwenLM/Omnilingua-Bench/tree/main/MuLA-Bench, releasing questions, reference answers, localized evidence and metadata. Source recordings were collected from public online media, and redistribution of the audio itself remains subject to source-specific permissions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only through captions and partially garbled OCR values (e.g., the Figure 4 heatmap grid); visual patterns were not independently verified.
- Appendix D.4 heatmap digits are garbled in extraction; cross-model SD claims rely on the surrounding prose.
- The GitHub release contents and per-source audio redistribution permissions could not be checked from the text.
- Model identifiers and dates are reported as printed (2026-era hosted models); no external verification of snapshots was possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.