Not Quite My Tempo: Voice Activity-aware Speech Synthesis for Lip-Synchronous Dubbing
Alejandro Pérez-González-de-Martos, Florian Lux, Angelina Elizarova, Milana Shkhanukova, Andreas Kellner, Mattia Antonino Di Gangi
eess.AS · cs.AI
Quick verdict
Binary VAD conditioning lifts inpainting-TTS frame alignment to ~96% for dubbing with no significant prosody cost, but no video-baseline comparison is made.
Summary
This Interspeech 2026 paper conditions an inpainting-style flow-matching TTS on a frame-level binary voice-activity (VAD) mask, extracted from the source-language reference audio with Silero VAD, to achieve lip-synchronous dubbing without paired video. The mask is embedded into the encoder representations of an F5-TTS-derived model; because inpainting operates on a fixed-length canvas, the source's voiced/silent pattern transfers directly to the target. Random masking of VAD embeddings during training (plus 20% conditioning dropout for classifier-free guidance) makes the condition optional at inference. Two models were trained: English-only on public LibriTTS-R and multilingual on public plus proprietary data, evaluated on 291 mTEDx utterances (7-15s, at least one pause over 500ms) with Greek, French, Portuguese and Russian sources. VAD conditioning raises Silero frame-alignment accuracy from 72.69% to 96.21% (LibriTTS model) and from 71.35% to 91.59% (multilingual), with consistent gains confirmed under Pyannote and TEN VAD. A subjective study (40 raters, 3840 ratings) finds no significant naturalness decrease (Mann-Whitney U, p>0.05), though WER rises (8.5% to 13.9%; 6.3% to 8.3%), attributed to poorly adapted translations. Main limitations: no comparison against cited video-conditioned dubbing systems, proprietary data/ASR for key components, and no code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Conditioning an inpainting flow-matching TTS on a frame-level binary VAD mask from the source audio raises frame alignment accuracy from 72.69% to 96.21% (LibriTTS model) and 71.35% to 91.59% (multilingual model), as measured by Silero VAD.
- VAD conditioning does not statistically significantly degrade pause-placement and prosody naturalness (MOS deltas of 0.01-0.13; pairwise Mann-Whitney U, p>0.05).
- Randomly masking VAD embeddings during training makes the condition fully optional at inference, so a single model lets editors enforce or relax lip-sync constraints.
- The observed WER increase under conditioning stems from translations that do not fit the temporal budget, and precision remains high when translations are isochronously adapted.
Strengths
Lightweight, video-free conditioning signal
Section 2.2 replaces lip-video embeddings with a frame-level binary VAD mask added to encoder representations; training needs no paired video and the system is decoupled from brittle face detection (Section 1 argues this matters for cartoons, anime, multi-speaker scenes).
Large, triple-verified alignment gains
Table 1: Silero frame accuracy rises from 72.69% to 96.21% (LibriTTS model) and 71.35% to 91.59% (multilingual), with per-language gains of 20-29 points; Figure 3 replicates the gap under Pyannote and TEN VAD, mitigating evaluator circularity.
Optional conditioning via random masking
Unusually honest failure-mode analysis
Detailed, partly reproducible recipe
Weaknesses
No external baseline comparison
Substantial WER cost explained only anecdotally
Silero-on-Silero circularity in headline numbers
Thin statistical treatment of the subjective study
Proprietary components limit verification
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This Interspeech 2026 paper (AppTek GmbH) targets automatic lip-synchronous dubbing: generating target-language speech whose voiced/silent pattern matches the source clip's timing. Rather than conditioning TTS on lip-video embeddings (Neural Dubber [11], DubWise [12], SyncVoice [13]), the authors condition an inpainting-style flow-matching TTS on a frame-level binary VAD mask extracted from the *source-language reference audio* using Silero VAD. Because inpainting synthesis operates on a fixed-length canvas, the source's activity pattern transfers frame-for-frame to the target. Three claims are made: (1) the model follows the VAD condition with high accuracy (Table 1: 96.21% Silero frame accuracy for the LibriTTS model vs 72.69% unconditioned; 91.59% vs 71.35% for the multilingual model); (2) prosody and pause placement do not significantly degrade (Table 2: MOS deltas of 0.01-0.13, pairwise Mann-Whitney U, p>0.05); (3) random masking of the condition during training makes it fully optional at inference, letting editors enforce or relax lip-sync with a single model.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How does VAD-conditioned TTS for dubbing differ from lip-reading (video-conditioned) approaches?
Instead of encoding lip movements from the video, this method gives the synthesizer a frame-level binary voiced/silent mask derived from the source audio with a VAD model (Silero). Training needs no paired video, and because the inpainting-style synthesizer works on a fixed-length canvas, the source's silence pattern transfers directly to the dubbed track. Reported frame-alignment accuracy reached 96.21% versus 72.69% without conditioning for the LibriTTS-trained model.
Does forcing lip-sync timing hurt the naturalness of the synthesized speech?
According to the paper's subjective study with 40 raters, pause placement and prosody naturalness did not decrease significantly (MOS differences of 0.01-0.13 on a 5-point scale, Mann-Whitney U p>0.05). However, word error rate rose (from 8.5% to 13.9% for the LibriTTS model and 6.3% to 8.3% for the multilingual model), mainly when the translated text did not fit the source timing, causing deletions, repetitions, or reordering.
Is code or data available for this paper?
No code release is announced. The core English model was trained on public LibriTTS-R data specifically for reproducibility, and the VAD tools (Silero, Pyannote, TEN VAD) are public, but the multilingual model uses proprietary data, WER evaluation used an internal proprietary ASR, and the 291-sample test list from mTEDx is not published. A demo page with audio examples exists (anondemos.github.io/NotQuiteMyTempo).
What data was used to evaluate the lip-sync dubbing model?
The evaluation set contains 291 utterances from the Multilingual TEDx corpus, each 7-15 seconds long with at least one pause exceeding 500 ms, balanced across four source languages (Greek, French, Portuguese, Russian), dubbed into English. A random subset of 25 samples was used for the subjective study with 40 English native speakers on the Prolific platform.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3-5 are only available as captions in the extracted text; per-sample distributions (onset/offset offsets in Fig 4, rating histograms in Fig 5) could not be inspected numerically.
- The demo audio examples (footnote 1) are not auditable in this review, so qualitative claims about pause placement rest on the paper's description.
- Some implementation details are absent from the 4-page format (e.g., the VAD-embedding masking probability, multilingual data composition), limiting full reproducibility assessment.
- This is the arXiv v1; the camera-ready Interspeech version may differ slightly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.