Auditory Illusion Benchmark for Large Audio Language Models
Hayoon Kim, Eunice Hong, Kyogu Lee
cs.SD · cs.AI
Quick verdict
First auditory-illusion benchmark for LALMs plus human baselines; sound design, but a 20-person absolute-pitch panel and no inferential stats limit generalization.
Summary
The paper introduces AIB, presented as the first benchmark of auditory illusions for Large Audio Language Models (LALMs). It compiles ten classic illusions spanning music, sound, and speech — missing fundamental, Shepard tone, Zwicker tone, phonemic restoration and others — labeled physics-based or physics+knowledge-based after Gregory's taxonomy, with 10,385 illusion and 4,444 matched control stimuli (a 1,566-stimulus mini subset). Models answer forced-choice questions and are scored with three descriptive metrics: Human-Likeness Accuracy (HLA), Reality Alignment (RA), and the Illusion Susceptibility Index (ISI = HLA − RA), benchmarked against a controlled listening study with 20 absolute-pitch participants aggregated by majority voting. Results: no model approaches the human profile. Audio Flamingo 3 achieves the best average ISI (0.455), yet a degenerate strategy that always answers 'illusion' would score 0.799, and several models fall below chance on control stimuli. Gemini 3.1 Pro and Kimi-Audio-Instruct are signal-faithful on physics-based illusions but more human-like on knowledge-driven ones, implicating language-level priors. Merely rewording prompts shifts ISI by up to 0.6 for three re-tested checkpoints. Limitations include the small, specialized human panel, absence of confidence intervals or significance tests, exclusion of unparseable model outputs, and a proprietary model whose scores may be hard to reproduce.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- AIB is the first benchmark of auditory illusions for LALMs, spanning ten illusions across music, sound, and speech with physics vs physics+knowledge mechanism labels.
- No evaluated LALM matches the human perceptual profile; the best average ISI is 0.455 (Audio Flamingo 3) against a human reference of 1.0.
- Several models (Gemini 3.1 Pro, Kimi-Audio-Instruct, Fun-Audio-Chat) flip from signal-faithful on physics-based illusions to human-like on knowledge-driven ones, suggesting alignment via language-level priors rather than shared low-level auditory processing.
- High ISI alone is not perception: a degenerate always-illusion strategy scores 0.799 average ISI while failing physics controls, so control accuracy must be read jointly.
- Prompt wording alone shifts ISI by up to 0.6, so measured illusion susceptibility is a property of the model-prompt pair.
Strengths
Genuine gap, first-mover scope
Section 2 documents that vision has illusion benchmarks (IllusionBench, HallusionBench, VLM illusion grounding) while audio evaluation stays recognition-anchored (SUPERB, AudioBench, MMAU); AIB fills this with ten illusions across music/sound/speech and an explicit mechanism taxonomy (3 physics, 7 physics+knowledge, Table 1).
Matched-control design with anti-gaming analysis
Every illusion has paired control stimuli, and Section 5 quantifies the degenerate baseline: always answering 'illusion' scores 0.799 average ISI while failing physics controls almost entirely, so ISI is explicitly read jointly with control accuracy rather than as a single leaderboard number.
Paired human baseline with a stated rationale
Honest prompt-sensitivity disclosure
Large stimulus scale and open release
Weaknesses
Small, atypical human panel defines the 'human profile'
No inferential statistics anywhere
Single-prompt protocol despite demonstrated 0.6-level sensitivity
Unquantified exclusions for unparseable outputs
Human-validation logistics unreported
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Auditory Illusion Benchmark for Large Audio Language Models (AIB)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Auditory Illusion Benchmark (AIB)?
AIB is a benchmark of ten classic auditory illusions spanning music, environmental sound, and speech (e.g., missing fundamental, Shepard tone, Zwicker tone, phonemic restoration), each with matched control stimuli — 10,385 illusion and 4,444 control stimuli in total. It is used to test whether large audio language models misperceive audio the way humans do, and the paper states it is publicly available at github.com/gillosae/aib.
What did the paper find about LALMs and auditory illusions?
No model matched the human response profile. Most models stayed faithful to the physical signal on physics-based illusions, while several (e.g., Gemini 3.1 Pro, Kimi-Audio-Instruct) became more human-like on illusions involving linguistic or musical knowledge. The best average Illusion Susceptibility Index was 0.455 (Audio Flamingo 3) versus 1.0 for the human reference, and a degenerate strategy of always answering 'illusion' would score 0.799 — so control-stimulus accuracy must be read jointly with susceptibility.
Who were the human participants and why absolute pitch?
20 participants, all possessing absolute pitch, judged randomized illusion and control trials without feedback through a web interface, instructed to use headphones in quiet environments. Absolute-pitch listeners were chosen because illusions like the Shepard and Tartini tones require reliable note-level judgments, which separates perceptual susceptibility from pitch-labeling error. Responses were aggregated by majority voting.
Can the benchmark results be reproduced?
Partially. The benchmark stimuli and their generation code are stated to be open source, and the open-weight model rows (e.g., Audio Flamingo 3, Qwen2-Audio) are re-runnable. However, the Gemini 3.1 Pro row depends on a proprietary API whose version and query conditions are not disclosed, and the paper does not include the exact prompt strings or output-parsing rules, which its own experiment shows can shift scores substantially.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 2's dense numeric grid was flattened by PDF extraction; cell values were read from the flattened text. Rows and columns are internally consistent (e.g., stated AF3 average ISI 0.455 matches its row), but individual digits could in principle carry extraction errors.
- The stated GitHub repository (gillosae/aib) was not inspected — network access was disabled for this review; code/data availability and reproducibility judgments rest on the paper's claims only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.