Clean Accuracy Does Not Guarantee Provenance Robustness: A Prospective Codec-Stress Evaluation of Audio Attribution
cs.SD · cs.LG
Quick verdict
Preregistered study shows clean audio-attribution accuracy collapses after codec transport; rigorous and honest, but artifacts not yet released.
Summary
An independent researcher preregisters and runs a codec-stress evaluation of closed-set audio provenance attribution. On ASVspoof 2019 LA (TTS-system identity) and ST-Codecfake-OOD (codec identity), with frozen WavLM-Base+, W2V2-BERT 2.0 and ECAPA-TDNN encoders plus linear and Proxy-Anchor heads, the author measures Macro-F1 loss after single-stage MP3, Opus, EnCodec and DAC transport. The analysis region was frozen from fidelity metadata (SI-SDR, PESQ-WB, STOI) before any attribution model was trained; all intervals use a component-level cluster bootstrap (45 components on LA, 28 on ST) with simultaneous max-|t| bands. Clean-qualified models (0.856-0.982 Macro-F1) lose up to 53.5 and 70.3 points at single in-support conditions; within one grid, losses range from near zero to over 50 points, encoder interactions exceed a ±5-point margin at 6 of 12 LA conditions and reverse across corpora; the registered matched-fidelity comparison was not estimable, and SI-SDR and PESQ-WB order conditions differently. A frozen 'overwrite' diagnostic derived from development data failed prospective confirmation. Main limitations: small effective sample sizes, five seeds realizing only three unique splits (duplicating configurations in the primary analysis), a post-registration Q1b support rule, and code and artifacts promised only for a future revision.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Clean attribution accuracy (0.856–0.982 Macro-F1) does not guarantee robustness: in-support codec conditions cost up to 53.5 and 70.3 Macro-F1 points for WavLM-Base+ on LA and ST respectively (Section V).
- Degradation is strongly condition- and representation-dependent: losses range from −0.4 to +53.5 points within one grid, and WavLM vs W2V2-BERT interactions exceed the ±5-point margin at 6 of 12 LA conditions with the ordering reversing across corpora (Section VI).
- No single fidelity scalar orders codec severity: the preregistered matched-fidelity comparison was not estimable, SI-SDR and PESQ-WB rank conditions differently (MP3-8k mid-grid on SI-SDR, last on PESQ-WB, yet most damaging), and non-codec controls depart from a fidelity-only account in opposite directions (Section VII).
- A development-derived 'overwrite' operational diagnostic, frozen and tested prospectively, was not confirmed — the paper explicitly claims measurement, not mechanism (Section IX).
Strengths
Genuine prospective registration discipline
Section III: the support mask was computed from fidelity metadata only (SI-SDR × STOI box, e.g. 5.8–15.4 dB × 0.866–0.991 on LA), materialized, hashed and committed before any attribution head was trained; hypotheses came from development data and were tested on separate prospective data, never pooled.
Null and negative results are preserved, not laundered
The registered matched-fidelity comparison is reported as 'not estimable' after failing its own gate (one admissible pair per corpus), and the frozen Q2 overwrite rule is reported as unconfirmed (2/4 named EnCodec cells missed, 3/6 DAC controls fired, Fig. 5) even though Section IX.E shows it looked compelling on development data.
Statistically careful inference at the right unit
Generality is bounded rather than asserted
Weaknesses
Artifacts promised, not present
Seed collisions duplicate weight in the primary analysis
Small effective sample sizes yield wide intervals
The primary estimand was lost and its replacement is post-registration
LA Macro-F1 averages across differing task sizes
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a preregistered measurement study asking one engineering question: how much of a clean audio-attribution score survives lossy transport? The scope is deliberately narrow and repeatedly restated: two closed-set provenance tasks (TTS-system identity on ASVspoof 2019 LA, attacks A07–A12 and A16; codec identity on ST-Codecfake-OOD), three frozen representations (WavLM-Base+ h4, W2V2-BERT 2.0 h3, ECAPA-TDNN), two heads (logistic regression; Proxy-Anchor), and a single-stage codec grid (MP3, Opus, EnCodec, DAC) plus four non-codec controls. Three contributions are claimed (C1–C3, Section I): clean accuracy does not guarantee robustness; degradation is condition- and representation-dependent; and no single fidelity scalar orders codec severity. No mitigation, mechanism, open-set or architecture-general claim is made — a scope discipline this paper actually honors.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does lossy audio compression break speaker/TTS attribution models?
In this study, often yes and sometimes drastically. Models that scored 0.86–0.98 Macro-F1 on clean audio lost up to 53.5 and 70.3 Macro-F1 points at specific single codec settings (e.g., MP3 at 8 kbit/s), while other settings in the same grid caused almost no measurable loss — the damage depends heavily on the exact codec, bit rate and encoder.
Which codec condition caused the largest attribution loss?
MP3 at 8 kbit/s caused the largest loss for WavLM-Base+ on both corpora (+53.5 points on ASVspoof 2019 LA and +70.3 on ST-Codecfake-OOD); for W2V2-BERT 2.0 the largest loss on LA was Opus at 6 kbit/s (+61.0 points). Notably, the largest losses fell at conventional codecs, not neural ones.
Is SI-SDR a reliable predictor of attribution robustness under compression?
No. The paper shows SI-SDR and the perceptual metric PESQ-WB rank codec conditions differently: MP3 at 8 kbit/s ranked mid-grid on SI-SDR but last on PESQ-WB while causing the largest attribution loss, and non-codec distortions deviated from a fidelity-only prediction in opposite directions.
Is the code or data for this paper available?
Not yet. The paper states that code, the registered protocol, frozen support masks with hashes, and per-cell results will be released in a future revision. The underlying corpora (ASVspoof 2019 LA, ST-Codecfake-OOD, MLAAD) are publicly available from their original distributors.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 are extracted only as axis-label, legend and caption text; visual content (interval shapes, scatter placement) could not be assessed from the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.