Building Trustworthy Mental Health Benchmarks on Bluesky: A Validation-Aware Weak-Supervision Framework
Gaurab Chhetri, Anandi Dutta, Subasish Das, Subasish Das
cs.AI
Quick verdict
Bluesky SI/MH weak-label benchmark with candid label audits; SI labels validate fairly well but MH labels (κ=0.222) are noisy, so MH scores track label noise.
Summary
The paper constructs two Bluesky benchmarks for suicidal ideation (SI) and mental-health disclosure (MH) detection. Posts were collected from the AT Protocol firehose (May–June 2025), filtered by task-specific lexicons, and weakly labeled by Llama-3-8B, yielding 8,346 SI posts (26.1% positive) and 9,988 MH posts (48.0% positive). Human validation on stratified subsets (276 SI, 250 MH) by two non-clinical annotators audits label quality: SI weak labels align reasonably with humans (κ=0.636, precision 0.947, recall 0.554, dominated by false negatives), whereas MH weak labels align poorly (κ=0.222, precision 0.276, recall 0.977, dominated by false positives from advocacy, metaphor, and third-person discussion). Transformer benchmarking shows protocol-dependent rankings: RoBERTa leads SI holdout F1 (0.792), BERT+LSTM leads SI 5-fold CV (0.916), and DistilRoBERTa leads MH CV F1 (0.905). The authors candidly note MH scores reproduce a noisy labeling function rather than the human construct. Main limitations: no statistical testing of small model gaps, models never evaluated on human-adjudicated labels, no stated code/data release, and small validation subsets.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A pipeline of AT Protocol firehose collection, lexicon retrieval, and Llama-3-8B weak labeling can produce sizable SI (8,346 posts) and MH (9,988 posts) benchmarks on Bluesky.
- Model rankings are protocol- and task-dependent: RoBERTa tops SI holdout (F1 0.792), BERT+LSTM tops SI 5-fold CV (0.9162), and DistilRoBERTa tops MH 5-fold CV (0.905).
- Weak-label noise is structured and task-specific: SI labels are dominated by false negatives, MH labels by false positives from advocacy, metaphor, and third-person discussion.
- Trustworthy decentralized-platform mental health benchmarks require joint evaluation of system design, label provenance, validation strategy, and deployment constraints.
Strengths
Label-provenance honesty is built into the evaluation
Table 1 is read against Table 7 explicitly: MH model F1 0.905 is framed as reproduction of a labeling function whose own positive-class F1 against human adjudication is 0.431 (§4.3, §5) — an unusually candid contrast rare in benchmark papers.
Task-specific noise characterization with verifiable numbers
Confusion counts are internally consistent (SI: 209 TN/36 TP/29 FN/2 FP; MH: 97/42/110/1), yielding a clean, evidence-backed contrast: SI errors FN-dominated (recall 0.554, precision 0.947), MH errors FP-dominated (precision 0.276, recall 0.977; κ=0.222).
Validation-protocol sensitivity is demonstrated, not asserted
Consumer-hardware efficiency reporting tied to governance
Underexplored decentralized-platform setting with systems framing
Weaknesses
Models are never evaluated on the human-adjudicated labels
MH benchmark fails construct validation yet still headlines results
No inferential statistics despite near-tied rankings
No stated code, lexicon, prompt, or data release
Thin and non-clinical validation layer; cross-corpus overlap unquantified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper presents an end-to-end system for building suicidal ideation (SI) and mental-health disclosure (MH) benchmarks on Bluesky via the AT Protocol firehose (May–June 2025). The pipeline runs lexicon-based candidate retrieval → Llama-3-8B weak labeling (local Ollama, deterministic binary outputs) → human validation of stratified subsets → transformer benchmarking. Two corpora result: 8,346 SI posts (2,180 positive, 26.1%; 6,648 users) and 9,988 MH posts (4,793 positive, 48.0%). Central claims: (1) decentralized platforms can host reproducible MH benchmarks; (2) model rankings depend on validation protocol and task; (3) weak-label noise is structured and task-specific; (4) benchmarks must be read with label provenance attached. The paper positions itself as a systems evaluation rather than a leaderboard — and largely honors that framing.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which model detects suicidal ideation best in this paper?
It depends on the evaluation protocol: RoBERTa-base achieves the best F1 on the 70/30 holdout split (0.792), while BERT+LSTM achieves the best F1 under stratified 5-fold cross-validation (0.916). The paper's point is precisely that the apparent winner flips with the validation protocol.
How reliable are the labels in this Bluesky mental health benchmark?
Mixed. Suicidal-ideation weak labels from Llama-3-8B agree reasonably with human adjudication (Cohen's κ=0.636, precision 0.947, recall 0.554 — misses are the main problem). Mental-health weak labels agree poorly (κ=0.222, precision 0.276, recall 0.977), so MH model scores mainly measure reproduction of a noisy labeling function rather than human-validated detection.
Is the dataset or code from this paper publicly available?
The paper only mentions a companion project webpage (ai-in-transportation-lab.github.io/mh-sui-hicss); it does not state a public release of the corpora, lexicons, prompts, or training code, so exact reproduction is currently not possible from the paper alone.
Where does the data come from and how much is there?
Public Bluesky posts were collected via the AT Protocol firehose between May and June 2025, filtered with suicide/mental-health keyword lexicons, then LLM-labeled. The SI corpus has 8,346 posts (26.1% positive); the MH corpus has 9,988 posts (48.0% positive). Human validation covered 276 SI and 250 MH posts.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 are captioned but their plotted content cannot be verified from the extracted text.
- The LLM labeling prompts, lexicon contents, and exact validation-sampling procedure are not included in the extracted text, so some methodology judgments rest on the paper's summaries.
- The companion webpage's content (code/data status) could not be verified from the text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.