Beyond the Flag: Clinical Framing Closes the Moderation Gap in Suicide Risk Measurement
Shreyas Krishnan, Gun Ahn, Jungjin Kim
cs.CL
Quick verdict
Claims moderation APIs flag but mis-measure suicide severity; an honest, well-controlled benchmark whose headline prompt-rubric circularity the authors themselves flag.
Summary
The paper reframes suicide-safety evaluation as an ordinal measurement problem. The authors release a benchmark of 516 r/SuicideWatch posts labeled by one licensed psychiatrist on a four-level C-SSRS-grounded scale (Indicator, Ideation, Behavior, Attempt); a 20-post reliability check found two further psychiatrists agreed within one tier (quadratic-weighted κ 0.938–0.957). They score vendor moderation APIs, zero-shot prompted LLMs, supervised baselines, and multi-agent aggregation under seven ordinal-aware metrics with bootstrap confidence intervals. Headline results: the best OpenAI moderation configuration separates low- from high-severity posts well (0.860 high-risk F1) but stratifies poorly (0.395 macro F1), over-predicting Attempt; a clinician-authored prompt lifts Claude Haiku 4.5 to 0.562 macro F1 (a stated 42% relative gain), though the authors concede the design cannot attribute this to the prompt rather than to using an LLM, since prompt and rubric share an author. Chain-of-thought hurts on Reddit but helps on a clinician-authored sentence set; multi-agent aggregation underperforms the best single model. Key limitations, disclosed candidly: single annotator, prompt–rubric shared authorship, a missing unspecified-prompt control arm, no non-self-harm control stratum, manually set moderation thresholds, and a smallest class of n=54.
Review complete
Assessment at a glance
4
risk flags identified
● high 2
● medium 2
● low 0
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Vendor moderation APIs separate low- from high-severity crisis posts well (OpenAI Weighted: 0.860 High-risk F1) but stratify ordinal severity poorly (0.395 macro F1, 0.604 QWK), systematically over-predicting Attempt.
- Zero-shot LLM classification under an expert-authored clinical prompt substantially outperforms moderation APIs (Claude Haiku 4.5: 0.562 macro F1, a stated 42% relative gain), and the paper concedes this comparison cannot isolate the rubric as the cause.
- Fine-tuning-style supervised baselines, added reasoning (CoT macro F1 0.435 on the best model), and naive multi-agent aggregation (0.542) all fail to beat simple zero-shot clinical prompting on this Reddit corpus.
- Reasoning's value depends on input register: CoT and GPT-5.5 hurt on long, noisy Reddit posts but help on short clinician-authored statements (CoT lifts Haiku accuracy 0.721→0.765; GPT-5.5 reaches 0.896 QWK).
- Graded severity rather than binary flagging is what a proportionate duty of care — and emerging regulation such as California SB-243 — requires of platforms.
Strengths
Hard-to-build expert-annotated benchmark with a transparent construction account
516 post-level r/SuicideWatch labels on a C-SSRS-grounded four-level schema (§3.1–3.2); the sampling frame, purposive screening criteria, class distribution (127/54/123/212) and the resulting no-negative-stratum consequence are all reported explicitly rather than buried.
Honest circumscription of every headline number
The paper repeatedly blocks over-reading: 0.860 High-risk F1 is labeled 'a severity split within suicide-related text, not detection' (§3.5, Limitations); the 42% gain is explicitly stated not to isolate the rubric as cause (Limitations); the 405-sentence set is 'transfer across register rather than external validation'.
Negative results reported as findings with inferential support
Genuinely ordinal, clinically anchored evaluation design
Weaknesses
Single-annotator reference standard with prompt–rubric circularity
The decisive ablation is missing and acknowledged as such
No base-rate-valid metric for the deployment story
Statistical reporting is thin relative to the number of comparisons
Result fragility at the smallest class
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks how well deployed safety signals recover **graded** clinical suicide-risk severity rather than a binary flag. It contributes: (1) a released (gated) benchmark of 516 r/SuicideWatch posts labeled by one licensed psychiatrist on a four-level, C-SSRS-grounded ordinal schema (Indicator/Ideation/Behavior/Attempt; §3.2); (2) a 'measurement problem' finding — OpenAI's moderation endpoint achieves 0.860 High-risk F1 but only 0.395 macro F1 (0.604 QWK), over-predicting Attempt (69/123 true Behavior and 21/54 true Ideation posts sent to Attempt, Appendix D); (3) the claim that expert-authored clinical framing, not fine-tuning, reasoning, or multi-agent aggregation, is the effective lever (Claude Haiku 4.5 zero-shot: 0.562 macro F1, 0.696 QWK, 0.591 accuracy; stated 42% relative macro-F1 gain); (4) a register-dependence result — CoT hurts on Reddit (0.435 macro F1, CI entirely below zero-shot) but helps on a 405-statement clinician-authored set (accuracy 0.721→0.765; GPT-5.5 best at 0.896 QWK). Claims are motivated by California SB-243's requirement of 'evidence-based methods for measuring suicidal ideation.'
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper 'Beyond the Flag' actually measure?
It measures how well deployed safety signals recover graded suicide-risk severity, not just binary flags. The authors built a benchmark of 516 r/SuicideWatch posts labeled by one licensed psychiatrist on a four-level C-SSRS-grounded scale (Indicator, Ideation, Behavior, Attempt) and scored vendor moderation APIs, prompted LLMs, and supervised baselines on identical items using seven ordinal-aware metrics. They emphasize it measures severity within suicide-related text, not detection against general traffic.
What is the 'moderation gap' the title refers to?
OpenAI's moderation API classifies low- vs high-severity crisis posts well (0.860 high-risk F1) but performs poorly at distinguishing adjacent severity levels (0.395 macro F1, 0.604 quadratic-weighted kappa), systematically over-predicting the most severe 'Attempt' category. A large language model given a clinically detailed prompt written by a psychiatrist reaches 0.562 macro F1 — though the authors caution their design cannot prove the prompt, rather than the LLM itself, is responsible.
Does chain-of-thought reasoning help or hurt suicide-risk classification here?
It depends on the input register. On the long, noisy Reddit posts, chain-of-thought prompting hurt the best model (Claude Haiku 4.5 fell from 0.562 to 0.435 macro F1). On a second set of 405 short, clinician-authored statements, reasoning helped: CoT raised accuracy from 0.721 to 0.765 and the reasoning model GPT-5.5 achieved the best results (0.896 QWK). The paper attributes this reversal to input register, while noting it cannot prove the attribution with certainty.
Is the benchmark dataset publicly available?
Not openly. It is released under gated, credentialed access: named researchers at identifiable institutions must sign a Data Use Agreement prohibiting redistribution, re-identification, and non-research use. The release contains post identifiers (not text), severity labels, the annotation rubric, all evaluated prompts, and corpus-reconstruction code; deleted posts will not rehydrate, preserving authors' ability to withdraw content.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.