Assessing Suicide Risk in Arabic Crisis Helpline Calls: A Comparison of Arabic and English Large Language Models
Linhai Ma, Rita El Hachem, Mahatab El Hajj, Lilian Ghandour, Samah Fodeh
cs.CL · cs.AI
Quick verdict
LLMs classify Arabic helpline calls' suicide risk from de-identified text (best macro-F1 81.19 AR, 85.00 EN); promising; tiny held-out tests limit certainty.
Summary
This study tests whether large language models can sort crisis-helpline calls into suicide-risk categories from transcripts alone, in Levantine Arabic — a setting almost absent from the NLP literature. Working with Lebanon's Embrace National Lifeline, the authors kept audio on site: calls were transcribed locally with a Whisper-family ASR model and de-identified with an Arabic NER model before any data left the service. Operators' C-SSRS ideation items were merged into two binary labels: at-risk (373 calls, 52.3% positive) and high-risk (297 calls, 30.0% positive). Transcripts were also machine-translated into English with Qwen2.5-72B, yielding paired corpora with identical splits. Eleven backbones — five instruction-tuned decoders fine-tuned with a uniform 4-bit QLoRA recipe and six BERT-family encoders — were evaluated on a single stratified 80:20 split. The best Arabic model (AceGPT-v2-70B) reached 81.19 macro-F1 and 90.61 ROC-AUC on high-risk; the best English model (Llama-3.3-70B) reached 85.00 and 92.59 with 88.9% sensitivity. High-risk calls separated more cleanly than lower-severity ideation, and encoders lost to decoders everywhere. The paper is unusually candid about trivial-classifier failures and metric pitfalls. Main limitations: tiny test sets (60–75 calls, 18 high-risk positives), one split without resampling, 442 of 987 calls unmatched to records, no ASR word-error-rate or translation-quality measurement, and labels taken from live operator assessments.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- De-identified Levantine Arabic helpline transcripts can be classified into C-SSRS-derived at-risk and high-risk categories by fine-tuned LLMs, with the best Arabic model (AceGPT-v2-70B) reaching 81.19 macro-F1 and 90.61 ROC-AUC on high-risk.
- Machine translation into English (Qwen2.5-72B) did not reduce best-observed performance: Llama-3.3-70B on translations reached 85.00 macro-F1, 92.59 ROC-AUC and 88.9% sensitivity for high-risk calls.
- High-risk calls separate more cleanly than lower-severity at-risk ideation, opposite to what class prevalence alone would predict (Section 3.4).
- Fine-tuned decoders beat fine-tuned BERT-family encoders on every task and language, and macro-F1/accuracy alone mask trivial-classifier failure modes, so positive-class recall and PR-AUC should be reported alongside them.
Strengths
Privacy-first within-service pipeline
ASR (Whisper-family, dialect-tuned decoding) and Arabic NER de-identification run on site and audio never leaves the helpline (Section 2.2); transcription/de-identification code is released. The authors present this as the first full such pipeline for a suicide helpline.
Clean paired Arabic/English design
Same calls, same call-level stratified splits, same labels on both corpora (Section 2.7); the 29 translation-incomplete transcripts were removed from both languages (Section 2.5), isolating the language effect in the translate-then-classify comparison.
Candid metric reporting
Uniform training budget across 11 backbones
One-directional missing-label rule
Weaknesses
Tiny single-split evaluation
Uncharacterized sample selection
Text transformations never measured
Primary-model selection optics
Label provenance
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Suicide-Risk Classification of Arabic Helpline Calls
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can AI assess suicide risk from Arabic crisis helpline calls?
This study fine-tuned large language models on 383 de-identified transcripts from Lebanon's Embrace National Lifeline. The best Arabic model reached 81.19 macro-F1 (ROC-AUC 90.61) for identifying high-risk calls, and an English-translated pipeline reached 85.00 macro-F1 with 88.9% sensitivity. The authors position the model as decision support for operators, not a standalone assessor, and results come from a small retrospective sample.
How was caller privacy protected in this study?
Audio recordings never left the helpline. Calls were transcribed on site with a speech recognition model for Levantine Arabic, then an Arabic named-entity recognition model removed names, places, organizations and other identifiers locally. Only redacted Arabic transcripts went to researchers, and phone-number blocks were stripped from filenames. Both the transcription and de-identification scripts are on GitHub.
Does translating Arabic crisis calls into English hurt classification performance?
In this dataset, no measurable loss was observed: the best English model on machine-translated transcripts (Llama-3.3-70B, 85.00 macro-F1 on high-risk) scored higher than the best Arabic model (AceGPT-70B, 81.19), possibly because English pretraining data is far larger. The authors caution this rests on one translation model, one sample, and unmeasured translation quality.
What are the main limitations of this suicide-risk classifier?
Test sets were tiny (60–75 calls, with 18 high-risk positives) from a single stratified split; 442 of 987 calls could not be matched to operator records and were excluded without comparison; ground truth is the live operator's C-SSRS entries with no reliability check; ASR word-error rate and translation quality were never measured; and the model predicts operator judgement, not actual suicidal behavior.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 are rendered plots; per-model values for all 11 backbones in this review were taken from the numeric Tables 4–8 and quoted text, not read off the figures.
- Three-epoch runs appear only via Table 8 and one quoted ROC-AUC/PR-AUC pair; full 3-epoch tables are not in the text, so all comparisons here rely on the 10-epoch results.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.