Cleaner Speech, Weaker Generalization: Revisiting Pitt-Derived Benchmarks for Alzheimer's Disease Detection
Luqi Sun, Shreeram Suresh Chandra, Lin Zhang, You-Jin Li, Brian MacWhinney, Yu Tsao, Emily Mower Provost, Berrak Sisman
cs.SD · cs.AI
Quick verdict
Argues denoised Pitt AD benchmarks boost in-domain scores but weaken generalization; evidence spans many models yet one small external test set.
Summary
This empirical study asks whether the speech-enhancement and curation steps behind the most widely used Alzheimer's disease (AD) speech benchmarks — Pitt, ADReSS, ADReSSo, and ADReSS-M, all derived from the original Pitt-origin recordings — actually help downstream detection. The authors quantify quality differences with DNSMOS (denoised variants score 3.18–3.60 BAK versus 2.70–2.73 for raw data), then train three supervised models (eGeMAPS features, XLS-R embeddings, and an SLS classifier over frozen XLS-R; five seeds, speaker-disjoint 80/20 splits) and probe five large audio-language models (LALMs), evaluating in-domain, cross-domain (an independent Lu corpus of 56 samples), and matched-versus-mismatched enhancement conditions using five enhancement methods. Supervised results show enhanced datasets raise in-domain Macro-F1 (SLS model: ADReSSo 0.8964 versus Pitt-origin 0.7910) yet reduce generalization to raw Lu speech (Pitt-origin 0.7449 versus ADReSS 0.4166); matching train/test enhancement helps but never beats the raw baseline. LALMs behave similarly: on denoised sets Kimi-Audio skews toward AD predictions (Pitt: AD-F1 0.7300 versus Control-F1 0.4548), while Qwen2-Audio and Audio Flamingo 3 collapse toward single classes. Limitations include one small external test corpus, unreported seed variance, and possible validation/test conflation for in-domain numbers. Code is released; data require gated DementiaBank access.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Speech-enhanced Pitt-derived datasets improve in-domain Macro-F1 but reduce cross-dataset generalization, with raw Pitt-origin models performing best on the Lu corpus (0.7449 versus ADReSS-trained 0.4166).
- Matching speech enhancement between training and test data alleviates but does not eliminate the penalty; no enhancement-based setting beats the raw-data baseline (0.7449).
- LALMs exhibit systematic prediction shifts on enhanced data — Kimi-Audio skews toward AD on denoised sets (Pitt: AD-F1 0.7300 vs Control-F1 0.4548) — and transcript input or two-shot examples do not correct the bias.
- Preprocessing sensitivity spans acoustic-feature models, self-supervised speech representations, and LALMs, so it is a broad modeling phenomenon rather than an architecture-specific artifact.
- Because real-world clinical speech is collected without standardized enhancement, curated 'cleaner' benchmarks are not necessarily more reliable for real-world AD detection.
Strengths
Controlled matched-enhancement design
Section 4.3 applies five modern enhancers (Denoiser, FRCRN, MossFormer, Resemble, MAP-SEMamba) symmetrically to Pitt-origin training data and Lu test data; matched always beats mismatched (e.g., MossFormer 0.6917 vs 0.6171) yet all settings stay below the raw-trained baseline (0.7449), cleanly separating train-test mismatch from enhancement-induced distribution shift.
Cross-architecture triangulation
Three supervised models spanning 6.2K–169K trainable parameters (eGeMAPS, XLS-R pooling, SLS layer selection over frozen XLS-R) plus five LALMs; the raw-data generalization gap (Pitt-origin 0.7449 vs ADReSS 0.4166 on Lu) reproduces across paradigms, supporting the not-architecture-specific claim.
Quantified preprocessing fingerprint
Honest significance annotation
Actionable negative result with a clean natural contrast
Weaknesses
All cross-domain claims rest on one small external corpus
Possible validation/test conflation for in-domain numbers
No variance, intervals, or multiplicity control
Enhancement confounded with curation in released benchmarks
Editorial and LALM-evaluation hygiene
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: Cleaner Speech, Weaker Generalization (arXiv 2609.00276)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
It finds that the denoised, curated versions of the Pitt Alzheimer's speech benchmarks (Pitt, ADReSS, ADReSSo) raise in-domain accuracy but hurt cross-dataset generalization: models trained on raw Pitt-origin recordings achieve the best score on the external Lu corpus (Macro-F1 0.7449), while ADReSS-trained models drop to 0.4166.
Does speech enhancement (denoising) help or hurt Alzheimer's detection models?
Both, depending on setting: it improves in-domain benchmark scores and DNSMOS background-noise quality, but it shifts the speech distribution so models generalize worse to raw, real-world recordings. Even when the same enhancement is applied to both training and test data, performance stays below the raw-data baseline.
Which AI models were tested?
Three supervised models (eGeMAPS acoustic features, an XLS-R self-supervised encoder, and an SLS layer-selection classifier) and five large audio-language models: Kimi-Audio, Qwen3-Omni, Qwen2-Audio, Audio Flamingo 3, and ultravox-v0.5-llama-3.2-1b.
Is the code and data available?
Code is released on GitHub (Sleepwalker554/Back_to_Origin). The datasets come from DementiaBank, which requires approved consortium membership and password-protected access; the enhancement pipelines behind the original Pitt/ADReSS/ADReSSo releases were never published.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction appears complete through Appendix G.4, ending with the final figure caption; no truncation was detected.
- Numbers cited from Figures 2–4 and the appendix figures were read from flattened figure-text extraction; a minor digit-transcription risk cannot be fully excluded.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.