arXiv paper· natural language processing / mental health informatics· shared-task system paper (empirical)

Snugi-AI-v2 @ eRisk 2026 Task 2: Early Depression Detection via a Learned Stopping Policy with Sustained Confidence Gate

Yuwen Chiu

cs.CL · cs.IR

Quick verdict

Claims first learned ERDE50 stopping policy for eRisk Task 2 with honest ablations; 'direct ERDE optimization' is BCE on heuristic labels, gains untested.

Summary

This CLEF 2026 working-notes paper presents Snugi-AI-v2, a submission to eRisk 2026 Task 2 (early depression detection from Reddit threads). The pipeline encodes each round with frozen MentalRoBERTa-base, maintains an O(1) incremental mean-pool over target-user posts, and classifies depression with a small MLP; timing is delegated to a learned stopping policy (a 5-32-16-1 MLP over trajectory features: probability, round, word count, delta-p, max p) plus a sustained confidence gate requiring N=3 consecutive high-confidence rounds. The authors claim this is the first stopping policy trained to directly optimize ERDE50, though training actually uses BCE loss on heuristic labels (first round with p>0.5). Five runs ablate two encoders, four stopping strategies, and three gate values on 909 eRisk 2025 subjects; 5-fold CV gives ERDE50 0.0290 vs 0.0369 (policy vs fixed threshold) with heavily overlapping CIs and no significance test. On the official test set (523 users, 500 rounds), best F1 is 0.73 (Run 1) and F_latency 0.70 (Runs 0/3), median alert round 8, and the system finished fastest (1h26m) among 16 complete-submission teams, ranking 9th by F1 but 4th on NDCG@100 (0.85). Negative results (GRPO, MentalLongformer, DeBERTa ensembling, BDI-II filtering) are documented. Main limitations: overstated 'direct ERDE optimization' framing, recall dropping below the fixed-threshold baseline despite the abstract's claim, mixed CV/validation scales in the ablation table, and an undefined DepRoBERTa in the encoder-reversal analysis.

Review complete

Assessment at a glance

natural language processing / mental health informatics

6

risk flags identified

● high 1

● medium 4

● low 1

Reproducibility readinessmedium
Code & data availableyes
Review confidencehigh
AI text signal0/20 passages flagged
Deep research viable

5 strengths · 5 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

  • A learned MLP stopping policy over 5-dim trajectory summaries outperforms fixed and tiered thresholds on ERDE50 (CV 0.0290 vs 0.0369; test 0.07 vs 0.08), and is the first such policy in eRisk Task 2.
  • The policy is trained to directly optimize ERDE50 end-to-end.
  • A sustained confidence gate (N=3 consecutive high-confidence rounds) reduces impulsive false alerts, adding -0.0023 CV ERDE50 over N=1.
  • The O(1) incremental pipeline completed the 500-round official evaluation in 1h26m, fastest among 16 complete-submission teams, with Run 3 reaching precision 0.80 and NDCG@100 0.85 (4th).
  • GRPO training, MentalLongformer encoding, DeBERTa ensembling, and BDI-II post filtering all failed to improve the system.

Strengths

  • Honest leaderboard self-positioning

    Table 7 shows Snugi-AI-v2 9th by F1 (0.73 vs HUGETIME 0.83) in full, with claims restricted to verifiable wins: fastest runtime (1h26m vs 2-6 days for peers), precision tie with HUGETIME (0.80, Run 3), and 4th-place NDCG@100 (0.85).

  • Rarely-reported negative results

    Section 5 documents GRPO collapse to all-alert, BDI-II filtering degrading ERDE50 from 0.0267 to 0.0548, MentalLongformer representation degradation, and DeBERTa ensembling showing no gain — concrete guidance absent from most shared-task papers.

  • Scalable O(1) design with stable behavior

  • Above-average reproducibility specification

  • Concrete failure-mode analysis

Weaknesses

  • Central claim overstates the training objective

  • No significance testing despite headline improvements

  • Mixed evaluation scales inside one ablation table

  • Abstract recall claim contradicted by own tables

  • Internal inconsistencies and an undefined model

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Preliminary Review: Snugi-AI-v2 @ eRisk 2026 Task 2

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

  • All 5 strengths & 5 weaknesses, analyzed in detail
  • 6 risk flags with severity, section & reviewer questions
  • Methods & statistical evidence assessment
  • 5 questions a peer reviewer would ask
  • Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is ERDE50 in eRisk early depression detection?

ERDE50 is the official eRisk cost metric: it averages a per-user cost that charges 1.0 for missing a depressed user, scales that penalty by a sigmoid latency term centered at round 50 for late alerts, and charges 0.1296 for a false alert on a control user. Alerting early on true positives is therefore much cheaper than alerting late.

Did Snugi-AI-v2 win eRisk 2026 Task 2?

No. By F1 it ranked 9th among complete-submission teams (0.73, versus 0.83 for top team HUGETIME, per the paper's Table 7). Its claimed wins are narrower: fastest runtime (1 hour 26 minutes), tied-top precision of 0.80 with Run 3, and 4th place on the ranking metric NDCG@100 (0.85).

What is the sustained confidence gate in this paper?

It is a rule requiring the learned stopping policy's fire probability to exceed a threshold (0.5) for N consecutive rounds before an alert is committed; N=3 was submitted for most runs. The paper's ablation shows N=3 gives the best ERDE50 (0.0267) for the target-only system versus 0.0274 at N=1, while the context variant does best at N=1 (0.0261).

Is code or data available for this paper?

The paper links a public code repository (github.com/chiuyuwen91/erisk-2026) and fully specifies hyperparameters and architecture. The eRisk 2026 data itself is not fully public: training reuses the eRisk 2025 test collection and the 2026 test labels are held by the task organizers, so access requires CLEF/eRisk registration.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

  • Figures are present only through their captions in the extraction; visual details (e.g., Fig 3a error bars, Fig 4 trajectory shapes) are assessed from captions and surrounding text, not the rendered plots.

Go deeper on this paper

Reproduce ResultsNot feasible for this paper — the core results cannot be re-derived by sandboxed code simulation.

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper
Snugi-AI-v2 @ eRisk 2026 Task 2: Early Depression Detection via a Learned Stopping Policy with Sustained Confidence Gate | Evolve.Pub