Psychoacoustically Aligned Latent Smoothing for Adversarial Robustness of Full-Duplex Speech-to-Speech Dialogue Models
Kian Shamsaie, Iman Modarressi
cs.SD · cs.AI · cs.CL · cs.HC
Quick verdict
Claims PALS cuts full-duplex speech agent hijack 91.7%→8.3%; rigorous and honest, but single-model evidence with LLM-judged outcomes.
Summary
The paper formalizes imperceptible adversarial attacks on full-duplex speech-to-speech dialogue agents as constrained optimization inside a psychoacoustic masking-threshold ball, with three goals: targeted semantic hijack, response suppression, and policy jailbreak. Against an undefended Moshi-style 7B agent, white-box attacks at Δ = 0 dB succeed in 91.7% (hijack), 88.4% (mute), and 76.2% (jailbreak) of trials. The proposed defense, PALS, injects anisotropic Gaussian noise at the RVQ latent interface with covariance from local codebook statistics, pairs it with masking-threshold-shaped input noise motivated by a maximum-entropy argument, and trains with a TRADES-style KL consistency term; no attack runs during training and inference cost is unchanged. The deployed PALS-Train cuts attack success to 8.3%, 11.2%, and 9.1% with UTMOS falling from 3.91 to 3.82, outperforming Gaussian augmentation, a Demucs front-end, input randomized smoothing, and PGD adversarial training under adaptive attacks, including simulated over-the-air conditions. A separate Monte Carlo-smoothed operator, PALS-Certify, yields an anisotropic certificate (47.0% of decisions certified at latent radius 0.5, protocol ceiling 0.616), which the authors honestly present as a floor at Δ ≈ −26 dB, far below the demonstrated attacks. Main limitations: outcomes other than mute are adjudicated by Gemini 2.5 Pro (κ = 0.88 on a 200-item audit), only one architecture with English corpora is tested, the input-space pullback of the certificate is heuristic, and no repository URL appears despite references to released code.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- White-box attacks confined beneath the psychoacoustic masking threshold break an undefended Moshi-style full-duplex agent at Δ = 0 dB with 91.7% (hijack), 88.4% (mute), and 76.2% (jailbreak) success.
- PALS-Train, a post-training defense with no inference-time cost, cuts these to 8.3%, 11.2%, and 9.1% at a 2.3% relative quality drop, outperforming Gaussian augmentation, Demucs purification, input randomized smoothing, and PGD-AT under adaptive attacks.
- Ablations attribute the defense to the psychoacoustic input dual and codebook-covariance latent noise (full 8.3% vs 12.7–14.9% for ablated variants), not to generic consistency training alone.
- PALS-Certify provides an anisotropic latent certificate (47.0% certified at radius 0.5, protocol ceiling 0.616) that the authors themselves bound at Δ ≈ −26 dB — a guaranteed floor far below the demonstrated attacks.
Strengths
Novel, well-posed threat model for full-duplex S2S agents
Section III ports the psychoacoustic masking threat (perceptual ball, Eq. 1, MPEG-1 model 1 thresholds) to an always-open duplex channel with three explicit goals; the mute attack uses a purely mechanical success criterion (agent activity < 5% of frames for 10 s), removing judge bias for that goal.
Adaptive-attack-aware evaluation
Attacks run 1,000 Adam iterations with 5 restarts and gradients through the straight-through quantizer, per the adaptive-evaluation guidance of [26][27]; OTA uses expectation over 120 pyroomacoustics rooms with babble; the Demucs purification baseline duly collapses (53.7% hijack), as adaptive-attack literature predicts.
Unusually honest certificate framing
Ablations isolate the actual drivers
Deployment-relevant cost profile
Weaknesses
Headline semantic metrics rest on one LLM judge
Single architecture, single language
Certified guarantee is largely symbolic for the threat
Reproducibility blockers
Shallow jailbreak success criterion
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper attacks and then defends full-duplex speech-to-speech dialogue agents (Moshi-style: streaming RVQ codec at 12.5 Hz, 7B temporal transformer, inner-monologue text stream). Section III formalizes a white-box attacker constrained to the psychoacoustic perceptual ball (Eq. 1, per-bin radii from MPEG-1 model 1) under three goals — targeted semantic hijack (Eq. 2), mute via silence tokens, and jailbreak via a universal adversarial prefix. Section IV introduces PALS, a post-training defense at the RVQ latent interface; Section V adds a certified smoothed variant. Headline claims, all verifiable against Table I: undefended success 91.7% ± 1.7 / 88.4% ± 2.0 / 76.2% ± 3.7 at Δ = 0 dB; PALS-Train 8.3% / 11.2% / 9.1% at UTMOS 3.82 vs 3.91 (a 0.09 absolute, "within 2.3%" drop); certified accuracy 47.0% at latent ℓ2 radius 0.5, capped at 0.616. The paper itself scopes the certificate as a floor at Δ ≈ −26 dB, below all demonstrated attacks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PALS and how does it defend full-duplex speech AI agents?
PALS (psychoacoustically aligned latent smoothing) is a post-training defense for speech-to-speech dialogue models. During training it injects Gaussian noise into the codec's latent space with covariance matched to local codebook geometry, adds matching input noise shaped by the psychoacoustic masking threshold, and ties both to the clean path with a KL consistency loss. The resulting model runs at inference with no extra cost and, in this paper, cuts adversarial attack success roughly tenfold at a 2.3% relative drop in audio quality.
How vulnerable are full-duplex voice agents like Moshi to inaudible attacks?
In this paper, white-box attacks constrained to the masking threshold of the carrier speech (Δ = 0 dB) achieved 91.7% targeted semantic hijack, 88.4% response suppression (mute), and 76.2% policy jailbreak against an undefended Moshi-style 7B agent, with about half of hijacks still succeeding over simulated over-the-air playback.
Does the paper's certified robustness guarantee protect against the demonstrated attacks?
No. The certified variant (PALS-Certify) guarantees decisions within a latent ellipsoid up to radius 0.616 (47.0% of decisions certified at radius 0.5), which corresponds to an input budget of about −26 dB — far below the −20 to 0 dB attacks demonstrated. The authors explicitly present the certificate as a guaranteed floor, with practical protection coming from empirical robustness of the deployed model.
What data and compute were used to train PALS?
Post-training used 3,118 hours of English dyadic conversational audio from five sources (otoSpeech, SSSD, CANDOR, Seamless Interaction, AMI), run for 80,000 AdamW steps on 32 H100 GPUs for roughly five days. Covariances were calibrated on 20 hours of held-out audio, and all semantic judgments used Gemini 2.5 Pro.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are available only as captions and partial embedded text; curve shapes and per-σ certified-accuracy values beyond labeled points (47.0%, 0.616) could not be visually verified.
- The extraction contains no repository URL despite references to 'released certify.py' and 'the released harness'; code availability is inferred from those mentions, not confirmed.
- Supplementary material (judge prompts, per-category jailbreak results, demographic breakdowns) is absent from the extracted text.
- arXiv v1 dated 23 Sep 2026; no peer-review status is known for this preprint.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.