Candidate Generation and Definition-Guided Verification for Sentence-Level Depression Symptom Recognition
Weiming Li, Catarina Barata, Miguel Constante, Joao Sanches
cs.CL
Quick verdict
Two-stage candidate+definition-verified depression symptom recognition; best scores on ReDSM-5 but no key margin is statistically significant.
Summary
The paper proposes a two-stage framework for sentence-level DSM-5 depression symptom recognition on the ReDSM-5 dataset (1,767 single-label sentences; post-disjoint 1,163/280/324 split via CP-SAT). Stage 1, MentalSBERT-S (all-mpnet-base-v2 with nine one-vs-rest heads and a triplet contrastive objective), selects one symptom candidate per sentence. Stage 2, a 4-bit QLoRA fine-tuned DeepSeek-R1-Distill-Qwen-14B, decides Present/Absent using the sentence, post context, and a candidate-specific DSM-5-informed definition, with an in-call self-check against that definition. On the 324-sentence test set the pipeline reaches 0.731±0.005 accuracy and 0.690±0.011 Macro F1, the best point estimates among encoder, NLI, MedGemma, prompted DeepSeek, and a matched single-stage supervised QLoRA baseline (0.698/0.661); Stage 1 alone hits 0.934 accuracy, matching fine-tuned generative inference without autoregressive decoding. Hierarchical bootstrap CIs reach significance nowhere: pipeline vs single-stage [−0.053, 0.105] (p=0.498); Stage-2 ablation trends are likewise inconclusive. A stratified, population-weighted audit by one psychiatrist (a coauthor) of 32 explanations yields 3.15/5 DSM-5 alignment and 2.52/5 usefulness, far lower for incorrect decisions (2.11/1.90). Limitations: small long-tailed test set (PSYCHOMOTOR: 6 instances), Absent F1 ≈0.45, single coauthor rater, unrecoverable Stage-1 errors, no stated code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The two-stage pipeline (MentalSBERT-S candidate generation + fine-tuned definition-grounded DeepSeek verifier) attains the best accuracy (0.731±0.005) and Macro F1 (0.690±0.011) of all evaluated methods on the ten-class end-to-end task, including a matched single-stage supervised QLoRA baseline.
- MentalSBERT-S matches the accuracy of a directly fine-tuned generative DeepSeek classifier (0.934 vs 0.918) on the nine-class Stage-1 task without autoregressive inference.
- Adding the DSM-5 definition alone reduces Stage-2 Macro F1 (0.630±0.052 vs direct 0.656±0.019), while adding the self-check restores and slightly exceeds it — reported as a non-significant trend.
- Explanation quality is governed primarily by prediction correctness (population-weighted audit: 3.52/2.73 for correct vs 2.11/1.90 for incorrect decisions).
- Performance remains limited for rare categories (PSYCHOMOTOR) and the sparse Absent class across all evaluated methods.
Strengths
Statistically honest reporting
Hierarchical bootstrap CIs (10,000 resamples, seeds + instances) are reported for every key contrast, and non-significant results are explicitly labeled 'trends' (Secs. IV-B/C/D, V): end-to-end vs single-stage [−0.053, 0.105] p=0.498, Stage-1 vs MentalRoBERTa p=0.334, verifier vs RAG p=0.665.
Matched single-stage baseline isolates the decomposition effect
The ten-class QLoRA baseline (Table 13: 0.698±0.003 acc / 0.661±0.015 Macro F1) shares the DeepSeek-R1-Distill-Qwen-14B backbone, training partition, and comparable optimization with the proposed pipeline, making the 0.731/0.690 comparison a genuine architecture test rather than a capacity confound.
Exceptional reproducibility detail
Self-correcting clinical audit design
Practical efficiency of the encoder-first design
Weaknesses
No headline result is statistically significant
Rare-class and Absent-class collapse
Thin, non-independent explanation-quality evidence
Preprocessing removes the hardest cases
No comparability with prior ReDSM-5 results
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper targets **sentence-level DSM-5 depression symptom recognition** on ReDSM-5 (derived from DepreSym), positioning itself between post-level screening systems and fine-grained symptom annotation (Sec. I). The proposed framework decomposes the task into Stage 1 candidate generation (MentalSBERT-S, a contrastively fine-tuned all-mpnet-base-v2 encoder with nine one-vs-rest sigmoid heads) and Stage 2 definition-grounded binary verification (a 4-bit QLoRA-adapted DeepSeek-R1-Distill-Qwen-14B that judges Present/Absent against a candidate-specific DSM-5-informed definition with an in-call self-check). The headline claims are that the pipeline attains the best accuracy and F1 of all evaluated methods — including a matched single-stage supervised baseline — and that its rationales align moderately with expert annotations. The paper is careful to frame outputs as annotation-level evidence markers, not diagnoses (Secs. III, V).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What dataset does this paper use?
ReDSM-5, a publicly released Reddit-derived dataset that links individual sentences to DSM-5-aligned depression symptom labels (built from DepreSym). The authors use 1,767 single-label sentences across nine symptom categories, split post-disjoint into 1,163 train / 280 dev / 324 test sentences.
What accuracy does the proposed method achieve?
On the 324-sentence test set: end-to-end 0.731 accuracy and 0.690 Macro F1 over ten classes (nine symptoms plus NO_SYMPTOM); Stage 1 alone reaches 0.934 accuracy over nine symptom classes. The authors report that none of these margins over the strongest baselines is statistically significant at this test-set size.
Is this system a depression diagnostic tool?
No. The authors explicitly frame outputs as sentence-level research annotations of symptom evidence, not diagnoses: DSM-5 duration, severity, and functional-impairment criteria are out of scope, and the paper states the system is not a substitute for professional clinical assessment.
Is code available for this paper?
The paper does not mention a public code release. However, its appendix is unusually complete: exact package versions, all training hyperparameters, the full Stage-2 prompt template, the nine candidate-specific DSM-5 definitions, and decoding settings, which together would allow reimplementation on the public ReDSM-5 dataset.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal values (confusion matrices, Figs. 2–4) are not extractable from text; only cell counts and metrics quoted in prose or captions were used.
- The text extraction appears complete (it ends with author biographies and references), so no truncation-based caveat applies.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.