Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models
Xize Cheng, Xize Cheng, Wenxu Jia, Chenyuhao Wen, Dongjie Fu, Zehan Wang, Xinyu Zhang, Xinyu Zhang, et al.
cs.CL · cs.HC
Quick verdict
Claims first provenance-annotated long-context speech benchmark; recency effect solid, but LLM-built QA, LLM judging and partial data release limit rigor.
Summary
The paper introduces Vox-Infinity, a benchmark for evaluating long-context understanding in spoken language models. Audio history is extended along two axes — turn count and turn duration — across three scenarios: ultra multi-turn dialogues built from SpokenWOZ (>20 turns), personal monologues from GigaSpeech (>2 minutes), and beyond-semantic dialogues from IEMOCAP, where neutral text is synthesized with Qwen-TTS into distinct emotional styles to disentangle acoustic emotion from lexical cues. Every QA pair carries answer-provenance annotations, and samples are grouped by 'required context duration' — the span from the earliest supporting evidence to the query. The benchmark totals 1,474 sessions, 13,516 QAs, and 127.96 hours. Seven spoken language models (Qwen3-Omni, GLM-4-Voice, AudioFlamingo3/Next, MiMo-Audio, Gemini-3.1-Pro, Qwen3.5-Plus) are evaluated with text (SenseVoice transcripts) or audio histories under a 32,768-token window, with GPT-4o as judge. Accuracy declines as evidence recedes: on ultra multi-turn dialogues the model average falls from 43.4% (0–2 min) to 38.2% (4–8 min); on beyond-semantic dialogues from 46.3% to 39.7%, and to 23.7% beyond eight minutes. Text history aids semantic retrieval (mean 71.3% vs 42.6%), while audio history wins for six of seven models on paralinguistic tasks. Limitations include GPT-4-generated questions judged by GPT-4o without reported human agreement, absence of significance testing, and only partial dataset release before acceptance.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Vox-Infinity is the first benchmark specifically designed to evaluate long-context understanding in spoken language models, providing answer-provenance annotations and required-context-duration grouping.
- Seven representative spoken language models exhibit a clear recency effect: accuracy declines as answer-supporting evidence lies farther back in the audio history (e.g., 43.4%→38.2% on ultra multi-turn; 23.7% beyond 8 minutes on beyond-semantic).
- Text history outperforms audio history for all seven models on semantic slot retrieval (mean 71.3% vs 42.6%), while audio history better preserves paralinguistic evidence, winning for six of seven models on beyond-semantic dialogues.
- On semantically controlled synthetic samples, audio history raises model-averaged accuracy from 23.8% to 30.4%, showing the modality gap on emotional content reflects acoustic rather than lexical cues.
Strengths
Length-aware evaluation design unique among audio benchmarks
Answer-provenance annotations plus grouping by 'required context duration' (0–2/2–4/4–8/8+ min bins, Table 1 'Prov.' column, Figure 2c) enable evidence-distance analysis that Table 1 shows VoxDialogue, ContextDialog, LA-Bench and BLAB all lack.
Clean disentanglement of acoustic emotion from lexical inference
Table 3 splits Beyond-Semantic into natural (n=2,758) and semantically controlled Qwen-TTS synthetic samples (n=2,622); natural samples show near-parity of audio/text averages (55.7% vs 56.3%) while controlled samples show a +6.6pp audio advantage (23.8%→30.4%), isolating prosody-dependent recognition.
Scale and interaction-structure coverage
Diagnostic dual-modality history comparison
Transparent, mitigated judging protocol
Weaknesses
LLM-mediated ground truth with unquantified judge reliability
No inferential statistics
Filter circularity in the controlled emotion subset
Partial data release gates independent verification
Provenance correctness rests on the generator's own labels
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Vox-Infinity (arXiv 2609.22452) positions itself as the first benchmark purpose-built for long-context understanding in spoken language models (SLMs). Construction scales audio history along two axes — turn count and per-turn duration — across three scenarios: Ultra Multi-Turn Dialogues (SpokenWOZ, >20 turns), Personal Monologues (GigaSpeech, >2 minutes reassembled from timestamps), and Beyond-Semantic Dialogues (IEMOCAP, plus Qwen-TTS syntheses of neutral text in distinct emotional styles). Its differentiator, per Table 1, is answer-provenance annotation and a "required context duration" metric (the span from earliest supporting evidence to the query), enabling length-binned evaluation (0–2/2–4/4–8/8+ minutes) that the compared audio benchmarks (VoxDialogue, ContextDialog, LA-Bench, BLAB) lack. Scale: 1,474 sessions, 13,516 QAs, 127.96 hours, average 36.38 turns/session. The central empirical claim is a recency effect: all seven evaluated SLMs degrade as evidence recedes — model-average accuracy on Ultra Multi-Turn falls 43.4%→38.2% (0–2 min→4–8 min); on Beyond-Semantic, 46.3%→39.7%, then 23.7% beyond eight minutes.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Vox-Infinity?
Vox-Infinity is a benchmark for testing whether spoken language models can understand long conversations. It contains 1,474 dialogue sessions with 13,516 questions over about 128 hours of audio, spanning multi-turn dialogues, long monologues, and emotion-focused conversations. Each question is annotated with the exact location of the evidence needed to answer it, so models can be scored by how far back in the conversation that evidence sits.
What is the main finding of the Vox-Infinity paper?
The paper finds a strong recency effect: spoken language models answer better when the needed evidence is recent and degrade as it moves back in the conversation. Average accuracy on emotion-focused dialogues falls from 46.3% to 39.7% and then to 23.7% when the evidence is more than eight minutes old. Text transcripts help with factual retrieval (71.3% vs 42.6% average on multi-turn dialogues), while raw audio is better for recognizing emotions.
Which models were evaluated on Vox-Infinity?
Seven spoken language models: Qwen3-Omni, GLM-4-Voice, AudioFlamingo3, AudioFlamingo Next, MiMo-Audio, Gemini-3.1-Pro, and Qwen3.5-Plus. Each was tested twice — once with the conversation history given as raw audio and once as an automatic transcript — within a 32,768-token context window, with GPT-4o grading the answers.
Is the Vox-Infinity dataset publicly available?
Partially. Example cases are available at the project page (vox-infinity.github.io), but the authors state that the full dataset and evaluation tools will only be released after the paper is accepted, so independent verification of the complete benchmark is not yet possible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Large chunks of the extracted text are omitted (marked '[...section omitted...]'), including the full contents of Table 2 and parts of Sections 3.2–3.3; several headline numbers are quoted from prose rather than verified against the tables.
- Figures 1–3 and the topic word cloud are images; curve shapes, per-model trajectories, and legend details could not be inspected.
- The Stage 3 manual-verification protocol is truncated in the extracted text, so annotator count, coverage, and provenance-label error rates are unknown.
- Evaluated model names (e.g., Gemini-3.1-Pro, Qwen3.5-Plus) and cited works could not be cross-checked against external sources under the no-network constraint.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.