OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents
Ruixun Liu, Yuxuan Wang, Jiacheng Xie, Yuhuan You, Donghua Cai, Junming Lin, Xiong-Hui Chen, Zhifang Guo, et al.
cs.SD · cs.AI
Quick verdict
Introduces a real-world FOA benchmark (2,972 QA + 900 nav) and an FOA-augmented omni model; gains are real but modest, SOTA claims rest on its own benchmark.
Summary
The paper introduces OmniEchoBench, a benchmark for spatial audio-visual perception and sound-guided navigation, and OmniEcho, an omni-modal model built on Qwen3-Omni-30B-A3B that adds a compact (11.9M-parameter) first-order ambisonics (FOA) encoder. The benchmark contains six tasks over 197 real-world scenes (2,972 QA pairs) plus 900 navigation samples recorded in 30 real indoor environments (12,900 FOA recordings); ground-truth answers are derived deterministically from 2 Hz annotations of panoramic video. Training data come from a controllable FOA rendering pipeline over Seedance-generated videos and augmented VLN trajectories (363,193 examples). In three stages (SigLIP semantic alignment, query-conditioned localization, integration into the frozen backbone), the model reaches 28.5% overall QA accuracy versus 18.5% for its Qwen3-Omni backbone and 11.4% for SO-7B, and attains 16.2% SR / 11.5% SPL on sound-guided navigation, above Seq2Seq/CMA and binaural SoundSpaces but below InternVLA-N1 (17.8/16.7). On VLN-CE R2R it trails modern VLN methods substantially (22.2% SR vs 50.6% for InternVLA-N1). Limitations include low absolute accuracy, generous scoring tolerances (±45°, ±2 m), reliance on synthetic training data with a demonstrated sim-to-real gap (28.5% to 34.9% after fine-tuning on 1,000 real samples), absence of statistical significance reporting, and code/data not yet released.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- OmniEchoBench is the first unified benchmark for spatial audio-visual perception and sound-guided navigation, with real-world FOA recordings: 2,972 QA pairs over 197 scenes and 900 navigation samples across 30 environments.
- Adding a pretrained 11.9M-parameter FOA encoder alongside the frozen semantic audio pathway makes OmniEcho state-of-the-art on the benchmark's spatial QA, reaching 28.5% overall vs 18.5% for its Qwen3-Omni backbone and 11.4% for SO-7B.
- Sound-guided navigation with audio-only guidance (16.2% SR / 11.5% SPL on OmniEchoBench-Nav) approaches earlier text-guided VLN baselines, though it trails InternVLA-N1 and modern VLN systems.
- Spatial audio is a valuable embodied signal, but fine-grained localization and distance estimation remain open challenges (distance Pearson r = 0.27 during closed-loop navigation).
- Freezing the FOA encoder during Stage 3 preserves spatial representations, and fine-tuning on just 1,000 real samples raises overall QA accuracy from 28.5% to 34.9%, exposing a sim-to-real gap.
Strengths
First unified real-world FOA benchmark for perception + navigation
OmniEchoBench pairs 2,972 QA pairs over 197 scripted real recordings with 900 navigation episodes over 12,900 real FOA recordings in 30 scanned environments (§3, Tables 7-8), unlike SoundSpaces' rendered acoustics; the authors explicitly position this against prior sim-only audio-visual navigation.
Deterministic, transparent scoring
QA ground truth is computed from 2 Hz panoramic-video trajectories, and correctness uses fixed rules with stated tolerances (Table 13); the LLM (qwen3.7-flash) only normalizes free-text answers, keeping judging reproducible (§A.4.1).
Clean component ablations
Honest failure analysis
Unusually complete methods disclosure
Weaknesses
SOTA claim is benchmark-relative with a thin comparator set
Navigation parity claim overstates the evidence
No statistical inference anywhere
Generous, unsensitivity-analyzed scoring tolerances
Synthetic-data bias and unreleased artifacts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
OmniEcho (arXiv 2609.23407v2, cs.SD/cs.AI; PKU, Alibaba Token Hub, Tsinghua) makes three coupled contributions: (1) OmniEchoBench-QA — six spatial-audio tasks over 197 scripted real-world recordings (2,972 QA pairs: source direction 1,069; 3D localization 521; source motion 473; camera rotation 309; cognitive map 600, Table 7); (2) OmniEchoBench-Nav — 900 sound-guided navigation episodes across 30 scanned indoor environments, backed by 12,900 real FOA recordings on dense receiver grids (Table 8); (3) OmniEcho, a Qwen3-Omni-30B-A3B-based model that grafts an 11.9M-parameter FOA encoder onto the backbone via three stages (SigLIP semantic alignment → query-conditioned localization → projector + LLM tuning, §4). The abstract claims state-of-the-art spatial perception and navigation 'close to' traditional VLN. The first claim holds within the paper's comparator set; the second needs qualification (see Limitations).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is OmniEchoBench?
OmniEchoBench is a benchmark from Peking University and Alibaba for testing whether AI models can understand where sounds come from in space, using 4-channel first-order ambisonics (FOA) recordings. It contains 2,972 question-answer pairs over 197 real-world recorded scenes (testing direction, 3D location, motion, camera rotation, and map-based localization) plus 900 navigation tasks across 30 real indoor environments where an agent must find a sound source.
How well does the OmniEcho model perform?
On the authors' own benchmark it scores 28.5% overall accuracy on spatial audio-visual QA (versus 18.5% for its unmodified Qwen3-Omni backbone and 11.4% for SO-7B) and 16.2% success rate / 11.5% SPL on sound-guided navigation. That navigation result beats older text-guided baselines (Seq2Seq, CMA) but sits below InternVLA-N1 (17.8% SR), and on the standard VLN-CE R2R benchmark it reaches 22.2% SR versus 50.6% for the strongest compared modern method.
Is OmniEcho's code and data available?
The paper states that code and data will be released at github.com/PKU-VaLuE-Lab/OmniEcho, but as of this review (paper version v2, September 2026, marked 'Under review') the artifacts had not yet been published, so the reported results could not be independently verified.
What is FOA (first-order ambisonics) audio?
First-order ambisonics is a 4-channel spatial audio format (stored in the ACN/SN3D convention) recorded from a single point that captures sound arriving from all directions. A model reading FOA audio can infer the direction, and with more difficulty the distance, of a sound source relative to the listener — which is why the authors use it as the spatial signal for their embodied agent.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures are not analyzable from the extracted text; Figure 6 ablation numbers and Figure 7/10 values are taken from captions and in-text statements only.
- A few passages were truncated in extraction (parts of §3.3 figure captions, some references, and transitions between appendix sections), though all main tables and the key appendix protocols were captured.
- Code and data are stated as forthcoming ('will be available') but were not verifiable at review time.
- The paper is a preprint marked 'Under review' (v2, 23 Sep 2026); numbers may change before publication.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.