XSQ-AST: An Explainable Audio Spectrogram Transformer Framework for Localising Synthetic Speech Artifacts
Ben Heritage, Luca Resti, Mónica Villanueva Aylagas, Timothy Mehlenbacher, Konrad Tollmar, James Alfred Walker
eess.AS · cs.LG · cs.SD · eess.SP
Quick verdict
Post-hoc SQ-AST saliency localizes TTS artifacts; 40-listener test shows moderate agreement (overall ρ 0.444) but rests on only 30 handpicked clips.
Summary
The paper presents XSQ-AST, a post-hoc explainability framework that wraps the pretrained SQ-AST speech-quality transformer (scoring MOS, noisiness, discontinuity, colouration and loudness on a 1–5 scale) with WhisperX phoneme alignment and four saliency methods — Raw Attention, Attention Rollout, Attention Flow and a GradCAM adaptation for regression — to temporally localize synthesis artifacts without retraining. Saliency maps are smoothed into continuous distributions via kernel density estimation and discretized onto phoneme boundaries via Phoneme Discretized Saliency Maps (PDSMs) with six pooling options; the framework also exports system-level reports, KDE flatness measures and ASR-confidence diagnostics. Validation used 30 handpicked VoiceMOS 2022 samples and a 40-participant online listening test with headphone screening, where listeners highlighted troublesome regions per dimension. Aggregated listener KDEs correlated with model saliency (median Spearman ρ: Attention Rollout 0.444, GradCAM+ 0.403, Attention Flow 0.378 overall), with different methods winning per dimension — e.g., Flow for colouration (0.732), GradCAM+ for MOS (0.649). PDSM analysis reached up to 0.806 (noisiness and loudness, Rollout/median pooling). Per-participant AUC-ROC was modest (0.626–0.666 for the best methods). Main limitations: small, handpicked evaluation set; oracle selection of the best method/pooling combination per dimension on the test data; moderate effect sizes without confidence intervals or multiple-comparison control; no external baseline comparison.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- XSQ-AST produces temporally localized artifact diagnostics by wrapping pretrained SQ-AST with saliency extraction and WhisperX alignment, requiring no model retraining (§1–2).
- Model-derived temporal KDEs from Attention Rollout, Attention Flow and adapted GradCAM+ correlate with aggregated listener highlights, with overall median Spearman ρ of 0.444, 0.403 and 0.378 respectively (Table 1, §4.2).
- Different saliency mechanisms are best suited to different perceptual dimensions — e.g., Attention Flow for colouration (ρ = 0.732), GradCAM+ for MOS (ρ = 0.649) and loudness (participant-level AUC = 0.861), Attention Rollout for discontinuity (ρ = 0.358).
- Phoneme-level localization via PDSMs reaches median ρ up to 0.806 for noisiness and loudness with Rollout/median pooling, with the best overall combination at 0.634 (GradCAM+/ℓ1/sum; Table 2).
- SQ-AST scores reflect perceived artifact severity: MOS vs. mean perceived influence ρ = −0.638 (p = 0.0001), dimension-specific scores ρ = −0.373 (p = 0.042) (§4.1).
Strengths
No-retraining post-hoc design
XSQ-AST wraps pretrained SQ-AST [8] and WhisperX [9] as a configurable pipeline (score thresholds, saliency choice, report types; §2.1), contrasting with Kuhlmann et al. [7] who must modify the training objective to get frame-level scores — a practical advantage for adopters of existing checkpoints.
Human-grounded validation
A 40-participant online listening test (20F/19M/1 other; mean age 32.8) with headphone screening [23] and per-dimension highlight annotation (§3) grounds the saliency evaluation in human perception rather than intrinsic proxies only — a step most XAI-for-speech papers skip.
Negative controls behave as expected
Two granularities plus system-level reporting
Code released
Weaknesses
Tiny, handpicked evaluation set with circular selection criteria
Oracle selection of method/pooling winners in Table 2
Moderate effect sizes with thin uncertainty reporting
No external baseline
Unvalidated GradCAM adaptation premise
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This Interspeech-style paper (eess.AS) introduces XSQ-AST, a post-hoc explainability framework that wraps the pretrained SQ-AST speech-quality transformer [8] with WhisperX [9] phoneme alignment and five saliency variants (Raw Attention, Attention Rollout, Attention Flow, and GradCAM adapted for regression as GradCAM+/−) to temporally localize synthetic-speech artifacts without retraining (§1–2). Saliency maps over Mel time-frequency patches are smoothed into continuous temporal distributions via KDE and discretized onto phoneme boundaries via Phoneme Discretized Saliency Maps (PDSMs) [12] with six pooling options (§2.4). The framework also emits system-level reports: thresholded dimension scores, KDE flatness, ASR confidence diagnostics, and phoneme-pair histograms (§2.5). Central claims: (1) post-hoc saliency correlates with listener-annotated artifact locations; (2) different saliency methods suit different perceptual dimensions; (3) phoneme-level localization is achievable with appropriate pooling. Code is released (footnote 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is XSQ-AST?
XSQ-AST is an open-source explainability framework that localizes artifacts in synthetic (TTS) speech in time and frequency. It wraps a pretrained speech-quality model (SQ-AST) with phoneme alignment (WhisperX) and saliency methods such as Attention Rollout, Attention Flow and an adapted GradCAM, producing smooth importance curves over time and phoneme-level highlights without retraining the model.
Does XSQ-AST agree with human listeners?
In a 40-person listening test on 30 VoiceMOS 2022 clips, the smoothed model importance curves correlated moderately with the regions listeners highlighted: overall median Spearman correlation was about 0.38–0.44 depending on method (up to 0.73 for colouration with Attention Flow), and per-listener AUC-ROC was around 0.63–0.67 for the best methods — above chance but far from perfect agreement.
Which saliency method works best for which speech artifact?
The paper reports Attention Flow worked best for colouration and noisiness, GradCAM+ for MOS and loudness, and Attention Rollout for discontinuity. However, these winners were selected after the fact on a small, handpicked sample set, so treat them as preliminary guidance rather than established practice.
Is the XSQ-AST code available?
Yes — the paper links a public GitHub repository (github.com/luca-resti/synth-speech-eval). The code covers the full pipeline (SQ-AST scoring, saliency extraction, WhisperX alignment, PDSM/KDE reports); release of the listening-test annotations is not explicitly stated.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (Fig. 1 signal flow, Fig. 2 example output, Fig. 3 scatter) are present only as garbled embedded text; the visual quality of plots and the Fig. 3 relationship could not be inspected.
- The listening-test annotation data release is not stated; independently recomputing Tables 1–3 depends on obtaining it.
- Interspeech-format page limits compress the listening-test protocol (per-dimension instructions, rating anchors, exact highlight interface); reviewed as published.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.