Sound-based Multi-Person 3D Pose Estimation
Yusuke Oumi, Yuto Shibata, Go Irie, Akisato Kimura, Yoshimitsu Aoki, Mariko Isogawa
cs.CV · cs.AI · cs.LG · cs.RO · cs.SD
Quick verdict
First multi-person 3D pose estimation from sound, with new 6h dataset; architecture gains are consistent but modest (~106.5mm MPJPE) and single-environment.
Summary
The paper reports what the authors describe as the first attempt to estimate 3D poses of multiple people using only sound. They propose SoundMHPE, an encoder-decoder in which a pair of speakers emits time-stretched pulse signals recorded by a four-channel ambisonics microphone. An Acoustic Multi-scale Encoder builds log-Mel spectrograms at three STFT window sizes (L, 2L, 4L) to combine temporal dynamics with fine frequency detail, while a Temporal Pose Decoder assigns each person per-frame DETR-style queries that jointly model intra-person temporal dynamics and inter-person interactions. The authors also introduce the 6-hour AMP dataset: 432K frames of acoustics synchronized with OptiTrack ground truth for 15 subjects in single-, double-, and triple-person settings. Under three-fold cross-subject validation, SoundMHPE attains 106.5 mm MPJPE, 65.0 mm PA-MPJPE, and 0.43 PCKh@0.5, outperforming adapted baselines from Shibata et al. (121.7 mm) and Yan et al. (119.9 mm). Ablations credit both proposed modules, and the components transfer to the public Person-in-WiFi 3D benchmark (122.6 vs. 127.4 mm MPJPE). Main limitations: modest absolute accuracy, a single measurement room, purely qualitative unseen-environment testing, and no variance or significance reporting.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- This is the first method to estimate multi-person 3D poses using only acoustic signals.
- Multi-scale STFT encoding with windows L, 2L, and 4L is required to isolate subtle individual signatures from overlapped multi-person acoustics.
- A Temporal Pose Decoder with per-frame per-person queries and factorized motion/interaction attention disentangles individuals better than single-query DETR-style decoding.
- SoundMHPE outperforms adapted single-person and WiFi-derived baselines on all metrics on AMP (106.5 vs 119.9-121.7 mm MPJPE), and its components transfer to the WiFi modality (Table 7).
Strengths
Genuinely new task with a purpose-built dataset
First acoustic multi-person 3D pose dataset: 6 hours / ~432K frames, 15 subjects, single/double/triple-person sessions (72+24+24 min per group), OptiTrack ground truth at 20 fps with a 21-joint skeleton (Sec. 4.1, Fig. 6).
Every architectural choice is ablation-supported
Component ablation (Table 3: 106.5 vs 115.2 w/o AME, 116.5 w/o TPD), attention decomposition (Table 5), and window selection (Table 6: L,2L,4L at 106.5 vs 114.5-118.5 mm for alternatives) each isolate a claimed contribution.
Cross-modal validation on public data
Graceful multi-person degradation
Weaknesses
Accuracy is coarse relative to the framing
Single-environment evidence base
No statistical uncertainty reporting
Reproducibility not committed
Key inference mechanics unaddressed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This arXiv preprint (2609.04902, v1) claims the first multi-person 3D human pose estimation using only active acoustic sensing. Four contributions are stated in Sec. 1: the task formulation, the Acoustic Multi-scale Encoder (AME), the Temporal Pose Decoder (TPD), and the 6-hour AMP dataset with 432K synchronized frames. The abstract asserts the TPD "precisely reconstructs frame-wise individual poses" — language the measured accuracy does not support (see below). Scope is deliberately narrow: up to three concurrent people, one instrumented room (Fig. 6), one hardware pairing (Edifier ED-S880DB speakers, Zoom H3-VR ambisonics microphone).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can the 3D poses of multiple people be estimated using only sound?
This paper claims the first such system, SoundMHPE. A pair of speakers emits chirp-like time-stretched pulse signals, a 4-channel ambisonics microphone records them, and the model estimates 21-joint 3D poses for up to three people simultaneously, with about 106.5 mm mean per-joint error in a controlled indoor room. It works without cameras (usable in darkness and behind some occlusions) but is far coarser than camera-based pose estimation.
What is the AMP dataset described in the paper?
AMP (Acoustic Multi-person Pose) is a 6-hour dataset of about 432,000 frames synchronizing active-acoustic recordings (speaker pair plus Zoom H3-VR microphone) with OptiTrack motion-capture ground truth. It covers 15 subjects (12 male, 3 female) in single-, double-, and triple-person sessions collected in three subject groups, with poses labeled at 20 fps over 21 joints.
How accurate is sound-based pose estimation compared to camera-based methods?
In this paper the best configuration reaches 106.5 mm MPJPE, 65.0 mm Procrustes-aligned MPJPE, and a PCKh@0.5 of 0.43, meaning most joints land outside half the head-neck distance. That is substantially coarser than typical RGB-based multi-person pose estimators, so the current value is demonstrating feasibility of the modality rather than deployment-grade tracking.
Does the proposed architecture work for sensing modalities other than sound?
The authors transferred their encoder attention and temporal decoder to the public Person-in-WiFi 3D WiFi benchmark, improving MPJPE from 127.4 mm (the original single-frame model) to 122.6 mm, while a naive multi-frame extension of that model degraded to 161.0 mm. This suggests the query and attention design generalizes across modalities, though accuracy remains modest.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Qualitative results (Figs. 7-9) can only be assessed via their captions and the authors' descriptions; the rendered visual content is not verifiable from the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.