Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection
Willem Berner, Julio Cesar Cavalcanti, Kalle Åström, Gabriel Skantze
eess.AS · cs.CL · cs.SD
Quick verdict
Adds visual features to VAP turn-taking on 400h of face-to-face data; FAU dominate, gains significant but modest, dyad-disjoint evaluation still missing.
Summary
The paper extends the self-supervised Voice Activity Projection (VAP) model for turn-taking prediction with visual features drawn from a ~400-hour subset of the Meta Seamless Interaction corpus of dyadic face-to-face conversations (328h/41h/41h splits, conversation-disjoint but not dyad-disjoint). Four feature groups — gaze, head orientation, body/hand pose, and facial action units (FAU) — are integrated through a greedy sequence of design choices: cross-attention fusion over concatenation, Δ3 delta features, and early gates, yielding model M5. M5 lowers VAP loss from 2.166 (audio-only) to 2.100 and improves balanced accuracy on hold/shift (76.41→78.31%), short/long (84.11→85.26%), and pre-hold/pre-shift (63.90→65.14%), with paired-bootstrap confidence intervals excluding zero on all four metrics. Ablations show FAU are the most informative single group, while body and gaze provide complementary information; a zeroing analysis identifies JawDrop, BrowLowerer, and EyesClosed as the most influential units. An exploratory improvised-vs-naturalistic analysis finds significantly better loss on naturalistic data but non-significant differences in event accuracies. Main limitations: the non-dyad-disjoint split means scores may be an upper bound; absolute gains are modest; the gaze extractor is a Meta-internal tool; and no code release is mentioned.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Visual features improve turn-taking prediction over audio-only VAP on large-scale face-to-face data, with the best model (M5) significantly better on VAP loss and all three event tasks (Table 2).
- FAU are the most informative single visual feature group, extending the finding from mediated/video-conferencing studies to co-situated face-to-face conversation.
- Body and gaze features carry complementary, non-redundant information, since the all-features model outperforms FAU alone on several contrasts (Table 4).
- Predictive performance differs between improvised and naturalistic subsets: loss is significantly lower on naturalistic data, while event-accuracy differences are not significant (§5.3).
Strengths
Closes a real scale-and-setting gap
Prior multimodal VAP work used 7-hour NoXi subsets or video-conferencing data (CANDOR); this is the first evaluation on ~400h of co-situated face-to-face conversation, directly testing whether the FAU-dominance finding (Onishi 2023; O'Connor Russell & Harte 2025a) generalizes beyond mediated settings.
Statistical transparency above area norm
Paired bootstrap (1000 resamples of 694 test conversations) with 95% CIs reported for every model and feature-group contrast (Tables 2 and 4); the improvised-vs-naturalistic analysis reports non-significant event-metric CIs rather than hiding them (§5.3).
Complementarity analysis beyond single-group ablation
Candid limitations section
Weaknesses
Non-dyad-disjoint splits undermine headline numbers
Greedy architecture protocol under-specified and limited
Modest effect sizes with no practical-deployment evidence
Reproducibility gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether visual cues improve Voice Activity Projection (VAP) turn-taking prediction in face-to-face conversation, and which visual feature groups contribute most. It uses a ~400-hour subset of Meta Seamless Interaction (328h train / 41h val / 41h test, balanced improvised/naturalistic, conversation-disjoint but not dyad-disjoint). Four groups are studied: gaze, head orientation, body/hand pose, and facial action units (FAU). Central claims: (1) every multimodal variant beats the audio-only baseline, with M5 significant on VAP loss and all three event tasks (Table 2); (2) FAU are the strongest single group (Tables 3–4); (3) body and gaze add complementary information; (4) performance differs across improvised vs naturalistic subsets, significantly only for loss (§5.3). Claims are conservatively scoped and largely match the presented evidence.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Voice Activity Projection (VAP) model?
VAP is a self-supervised transformer model that predicts which of two conversation partners will be speaking over the next two seconds, from raw audio processed by a frozen CPC encoder. It is used to help spoken dialogue systems decide when to take or yield the conversational floor, replacing simple silence thresholds.
Do visual cues actually improve turn-taking prediction in this paper?
Yes, modestly but significantly. The best multimodal model (M5) reduced VAP loss from 2.166 (audio-only) to 2.100 and raised hold/shift balanced accuracy from 76.41% to 78.31% on about 400 hours of face-to-face conversation data, with bootstrap confidence intervals excluding zero on all reported tasks.
Which visual features matter most for predicting turn-taking?
Facial action units (FAU) were the most informative single group — especially JawDrop (AU26), BrowLowerer (AU4), and EyesClosed (AU43), which relate to speaking and timing. Body pose and gaze added smaller but complementary gains, and the model combining all feature groups performed best overall.
Can this multimodal turn-taking model be used in real-time dialogue systems today?
Not directly demonstrated. The paper evaluates offline prediction on clean, channel-separated audio, notes the multimodal model costs more to run than audio-only VAP, and argues (without testing) that gains might be larger under noisy single-channel conditions typical of deployed systems. The non-dyad-disjoint evaluation also means reported scores may be an upper bound.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 are described in captions but not visually analyzable from the extracted text; architecture details rely on §4 prose.
- The extraction appears to cover all main sections and tables; any appendix or supplementary material beyond the reference list is absent.
- arXiv v1 preprint (Sep 2026); peer-review status unknown.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.