Audio-Visual Turn-taking Prediction in Cocktail Party Scenarios
cs.SD · cs.CL
Quick verdict
First cocktail-party evaluation of turn-taking models: audio degrades up to 38% F1, fine-tuning helps video/MM but not audio; credible, but eval is dev-only.
Summary
The paper presents the first systematic evaluation of predictive turn-taking models in cocktail-party conditions, building a dyadic testbed from AVCocktail with an automated procedure that derives shift/hold labels from sentence-level transcriptions using a relaxed 2.0s threshold, manually verified on the dev set. Three Voice Activity Projection variants — audio-only, video-only, and multimodal MM-VAP — pre-trained on 602 hours of clean Candor videoconference data are benchmarked. All modalities degrade under overlap and noise: Audio-VAP falls from 81.05% to 49.83% weighted F1 (about 38% relative), MM-VAP from 82.57% to 56.61%, while Video-VAP stays stable (70.68% vs 69.23%). Fine-tuning on six hours of AVCocktail lifts MM-VAP by about 30% relative to 73.65% and Video-VAP to 73.01%, but Audio-VAP fails to adapt; all models show 1.6–5.6% catastrophic forgetting on Candor. Synthetic-noise augmentation at 0dB helps but trails real noisy-data training. A pre-training scale ablation (6–600h) shows adaptation gains grow with data for video and multimodal models while audio never adapts. Limitations: evaluation uses only the dev set (the official test split lacks transcriptions, and dev also drives model selection), mono audio is duplicated to stereo, training labels are not manually verified, and statistical reporting lacks effect sizes and multiple-comparison control. Code and labels are publicly released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PTTMs pre-trained on clean Candor degrade substantially on cocktail-party AVCocktail, up to 38% relative weighted-F1 drop (Audio-VAP: 81.05%→49.83%), with audio most sensitive and video most stable (70.68%→69.23%, p=0.3689).
- Fine-tuning on 6.15h of AVCocktail yields up to 30% relative improvement (MM-VAP to 73.65% weighted F1) and 5.46% for Video-VAP, but Audio-VAP fails to adapt (↓4.70%).
- Pre-training with 0dB synthetic noise augmentation improves cocktail-party robustness without harming the original domain, but underperforms training on real noisy data (MM-VAP: 60.55% vs 69.84%).
- Visual-supported models suffer less catastrophic forgetting (under 6% relative on Candor) and their adaptation gains grow with pre-training scale (6–600h), while audio-only models never adapt regardless of scale.
Strengths
First genuine cocktail-party evaluation of PTTMs
Section I and V position this against prior work that used only synthetic noise augmentation ([14],[15]); the AVCocktail-derived testbed (Section II-C) exposes models to real overlap, real background noise, and Lombard-adapted speech, with SNR analysis (Fig. 1: median ~15dB vs 90–100dB in Candor) documenting the domain gap.
Label-quality transparency
The appendix manually verifies dev labels with ELAN (Table III: 33 dyads, 1,916 shifts, 1,758 holds), documents failure modes (backchannels mislabeled as shifts, timestamp drift; Fig. 7), and shows in Table IV that model rankings are stable under corrected labels — e.g., MM-VAP Candor→AVCocktail rises from 73.65% to 78.43% weighted F1.
Comprehensive modality analysis across regimes
Honest negative results and released artifacts
Weaknesses
Dev-set-only evaluation with selection bias
Non-standard event definition without sensitivity analysis
Stereo input degeneracy confounds the audio conclusion
Thin statistical reporting
Training labels never manually verified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.17056, Sigmedia Group, Trinity College Dublin) asks two questions: (RQ1) do predictive turn-taking models (PTTMs) trained on clean, controlled data generalise to cocktail-party conversations — multi-party gatherings with overlapping speech, background noise, and Lombard-adapted speech production; (RQ2) does simple fine-tuning adapt them, and how does pre-training scale condition adaptation? Central claims, all tied to Table II: (1) Candor-pre-trained VAP models degrade up to 38% relative weighted F1 on the derived AVCocktail testbed (Audio-VAP 81.05%→49.83%); (2) fine-tuning on 6.15h in-domain data recovers performance for the multimodal model (56.61%→73.65%, ~30% relative) and video model (69.23%→73.01%) but not audio (49.83%→47.49%); (3) 0dB synthetic-noise pre-training (Group IV) helps but trails real noisy data (MM-VAP 60.55% vs 69.84%); (4) adaptation gains and forgetting scale with pre-training size across 6–600h (Fig. 6). The claim to be first in genuine cocktail-party turn-taking evaluation is, to my knowledge, correct.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a cocktail-party scenario in turn-taking research?
It is a setting where several conversations happen in the same room at once, so a recording contains overlapping speech, background noise, and competing visual cues. This paper builds a benchmark for it from the AVCocktail dataset (sessions of up to 8 speakers and 4 parallel conversations) and tests whether models trained on clean two-person conversations can still predict when a speaker's turn will shift or hold.
How much do noisy, overlapping conditions hurt turn-taking prediction accuracy?
In this study, models pre-trained on clean videoconference data lost up to 38% of their weighted F1 when tested on cocktail-party recordings — the audio-only model fell from 81.05% to 49.83%. The video-only model was far more stable (70.68% to 69.23%), and the audio-visual model fell in between (82.57% to 56.61%).
What is a VAP (Voice Activity Projection) model?
VAP is a Transformer-based architecture that, at each moment, predicts each speaker's voice activity over the next two seconds, divided into bins of 200–800ms. This lets a system anticipate whether the current speaker will keep talking (hold) or a turn shift will occur. The paper tests audio-only, video-only, and audio-visual VAP variants.
Is the code and data from this paper available?
Yes. The authors state that all code and the derived turn-taking labels are publicly available at github.com/lggvu/mm-turn-taking. The underlying corpora are AVCocktail and Candor; Candor requires a licence/request for access.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (1, 3, 5–8) are not analyzable from extracted text; Figure 6 values are known only approximately from caption and prose.
- Text appears complete through the appendix, but extraction cannot rule out minor omissions (e.g., exact axis scales, additional table formatting).
- arXiv metadata (2609.17056, dated Sep 2026) is taken as given and not independently verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.