Masked Autoregressive Speech Enhancement with Continuous Neural Audio Codec Representations
Yoto Fujita, Simon Leglaive, Laurent Girin
cs.SD · cs.AI
Quick verdict
Unifies AR/non-AR/masked decoding for speech enhancement on continuous codec latents with a credible compute-quality trade-off; evaluation is proxy-only and statistical evidence absent.
Summary
MARSE reformulates speech enhancement as iterative decoding of masked clean-speech frames in the continuous latent space of the DAC neural codec, adapting the masked-autoregressive (MAR) framework from image generation. Under an identical Conformer, codec, and training setup, the authors compare three decoding policies: causal block-wise decoding, non-causal decoding with random index selection, and a non-causal 'oracle' variant that selects frames using ground-truth clean speech. On Libri1Mix and a synthetic out-of-domain set (LibriDEMAND), MARSE with N=10 falls between the C-NAR and C-AR baselines in both quality and compute (1912 GFLOPs versus 1235 and 3856), and intelligibility improves markedly over C-AR (dWER 12.68 versus 20.89 in-domain; 9.35 versus 15.91 out-of-domain). Quality rises with the number of iterations, approaching both endpoints on a 300-sample subset. Main limitations: reliance exclusively on non-intrusive proxies (DNSMOS P.835, wav2vec2-based dWER), no reported statistical uncertainty, no listening tests, and the practical non-causal policy still trails the causal one in-domain. Code and audio examples are released online.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MARSE provides a unified probabilistic framework (Eq. 1-2) in which non-AR (N=1), block-wise AR, and frame-wise causal AR (N=T) decoding policies are special cases of one trained model.
- Under identical Conformer, DAC codec, and training setup, N=10 MARSE delivers quality and compute intermediate between retrained C-NAR and C-AR baselines (e.g., SIG 3.62, OVRL 3.34, 1912 GFLOPs in-domain), demonstrating a flexible trade-off.
- MARSE policies preserve intelligibility close to C-NAR where C-AR degrades severely (dWER 12.68 vs 20.89 in-domain; 9.35 vs 15.91 out-of-domain, Table 1).
- Oracle non-causal frame selection achieves the best out-of-domain results (dWER 8.86), suggesting a practical non-oracle confidence measure could improve generalization.
Strengths
Genuinely controlled policy comparison
All MARSE variants share one Conformer (16 blocks, hidden 384), one NAC (DAC, D=1024) and one training recipe (Sec. 3.1), so decoding policy is the only manipulated variable — the 'ceteris paribus' design the token-based literature lacks.
The framework subsumes the known endpoints
Eq. (1)-(2) is a proper chain-rule factorization over blocks: N=1 recovers non-AR prediction, N=T recovers frame-wise causal AR. Fig. 2 confirms MARSE-causal approaches C-NAR at N=1 and C-AR at N=50 (OVRL ~3.39-3.40 scale), empirically validating the unification.
Intelligibility result is practically meaningful
Honest oracle baseline and disclosures
Reproducibility support
Weaknesses
Evaluation rests entirely on non-intrusive proxies
No statistical uncertainty at all
GFLOPs accounting does not visibly reconcile
Practical non-causal benefit remains unproven
Single dataset family, single codec
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MARSE (Secs. 1-2) adapts the masked autoregressive (MAR) framework of Li et al. [14] to speech enhancement over blocks of continuous DAC latents with Gaussian conditionals, explicitly positioning itself as 'a unified framework encompassing AR, non-AR and masked AR decoding policies' rather than a new decoding method. N=1 recovers all-at-once non-AR prediction (C-NAR-like), N=T recovers frame-wise causal AR (C-AR-like). The paper's central claims: (i) the unification is sound and empirically continuous across N (Fig. 2); (ii) under identical Conformer/codec/training, N=10 MARSE yields intermediate quality and compute with markedly better intelligibility than C-AR (Table 1); (iii) oracle non-causal selection is best out-of-domain, motivating future practical confidence measures.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MARSE (masked autoregressive speech enhancement)?
MARSE is a speech enhancement method that works in the continuous latent space of the Descript Audio Codec (DAC). It treats enhancement as iterative decoding: at each of N iterations, a subset of masked clean-speech frames is predicted from the noisy signal and previously decoded frames by a Conformer. Depending on how many frames are decoded per iteration and in what order, the same framework behaves as non-autoregressive (N=1), block-wise autoregressive, or fully frame-wise causal autoregressive (N=T) enhancement.
How does MARSE compare to C-NAR and C-AR baselines?
In the paper's experiments (N=10 iterations, Libri1Mix and out-of-domain LibriDEMAND), MARSE sits between the two retrained baselines on both DNSMOS quality and compute: e.g., OVRL 3.34 at 1912 GFLOPs versus C-NAR's 3.32 at 1235 and C-AR's 3.37 at 3856 in-domain. On intelligibility (dWER), MARSE stays close to C-NAR (12.68 vs 12.84 in-domain) while C-AR degrades badly (20.89 in-domain, 15.91 out-of-domain).
Why use continuous codec representations instead of discrete tokens for speech enhancement?
The authors build on a prior finding (Kammoun et al., ICASSP 2026) that continuous NAC latents — encoder outputs before vector quantization — preserve acoustic details better than discrete tokens, improving speech quality and intelligibility. They cite this prior work rather than rerunning token-based baselines, since those were already shown to underperform the continuous C-NAR/C-AR models.
Does more decoding iterations always improve speech enhancement quality in MARSE?
Not indefinitely. The paper's Fig. 2 shows DNSMOS OVRL rising from N=1 to about N=20 and then stagnating in the 20-40 iteration range, which the authors identify as a good performance/compute trade-off. Note this trend was measured on a 300-sample subset of the test set using only the OVRL metric.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Text is complete from abstract through references; apparent mid-sentence line breaks are PDF extraction artifacts, not truncation. All cited figures/tables (Table 1, Fig. 2 axis ticks 1-50, footnotes) are present.
- Fig. 2 has no numeric labels, so plotted values are read approximately from axis ticks (3.37-3.41); statements about it are qualitative.
- Review is based on proxy metrics as reported; no external validation of DNSMOS/dWER claims was possible from text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.