Aligning with Lived Experience: Heterogeneous Benefits of Fine Tuning in Mental Health Support Generation
Mohit Chandra, Nabin Kim, Eli Min, Aamogh Sawant, Tanmay Sutar, Munmun De Choudhury
cs.CL · cs.AI · cs.CY
Quick verdict
COPES dataset + 3-axis eval show fine-tuning lifts community-aligned peer support >50% for general LLMs, but gains are uneven; LLM-made gold data caps claims.
Summary
This paper introduces COPES, a dataset of 4,455 peer-support-seeking Reddit posts from 55 mental-health subreddits (2011-2021, grouped into five categories), labeled by GPT-4.1-as-judge with four coping-strategy tags; each post carries a preferred response synthesized from community comments and an LLM-written rejected response built from an 11-type harmful-support taxonomy. Using a three-axis framework - embedding-based Strategy Alignment (cosine >= 0.8), Strategy Congruence, and Emotion & Tone via Empath and KL divergence - the authors evaluate Qwen 3-4B Instruct, Gemma-4-E4B, and MediPhi-Instruct under zero-shot, LoRA-SFT, and SFT+DPO. Post-training raises Strategy Alignment by over 50% in relative terms for the general-purpose models (Qwen 0.30 to 0.47-0.48; Gemma 0.32 to 0.50) but only 3-5% for MediPhi, and gains vary by subreddit and coping type, with meaning-focused posts improving most. DPO shifts outputs toward problem-focused advice while suppressing emotion-focused strategies and reduces topical similarity to references. Main limitations: the gold responses are themselves GPT-4.1 syntheses, human validation is small and author-only, only answered pre-2021 threads are covered, and no multiple-comparison control is reported.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Post-training on COPES improves community-grounded Strategy Alignment by over 50% in relative terms for general-purpose 4B models (Qwen 0.30 to 0.47-0.48; Gemma 0.32 to 0.50), but only 3-5% for MediPhi-Instruct.
- Alignment gains are heterogeneous across subreddit categories and coping types, with meaning-focused posts improving most (Gemma +74.27%, Qwen +69.44%).
- Post-training induces a distributional shift favoring problem-focused recommendations while suppressing emotion-focused strategies.
- SFT+DPO reduces topical alignment with reference responses across all models (simHDP; e.g., Gemma -32.5% for C1).
- Emotion & Tone alignment improves for Gemma (+13.51% SFT) and MediPhi (+3.78% SFT) but worsens non-significantly for Qwen (-2.33% SFT).
Strengths
Community-centered evaluation axes grounded in coping theory
The three metrics (embedding-based Strategy Alignment, Strategy Congruence, Empath/KL Emotion & Tone, Sec. 3) operationalize community alignment rather than generic clinical QA, with coping categories derived from Folkman's framework and strategy extraction validated at Cohen's kappa 0.90 over 187 strategies.
Honest reporting of negative and heterogeneous results
The paper foregrounds that SFT+DPO suppresses emotion-focused strategies, lowers simHDP topical alignment for all models (e.g., Gemma C1 -32.5%), leaves MediPhi nearly flat (+3-5%), and worsens Qwen's E&T KL — and explicitly flags its own C3 outlier (+340%/+397.78%) as an outlier at n=10.
Fully specified, identical post-training pipeline across models
Theory-grounded negative-response construction for DPO
Multi-stage human validation of an LLM-labeled dataset
Weaknesses
LLM-synthesized gold standard creates circularity risk
Statistical reporting is thin for the number of comparisons
Category-level conclusions rest on very small strata
Alignment metric's absolute ceiling is unexplored
Motivation-construction mismatch and author-only annotation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
COPES (COmmunity-centered Peer Engaged Support) is a dataset of 4,455 Reddit posts seeking non-clinical peer support, plus a three-axis evaluation framework (Strategy Alignment, Strategy Congruence, Emotion & Tone) used to test whether post-training steers small LLMs toward community-aligned peer support (RQ1) and how gains vary across coping-strategy labels and subreddit categories (RQ2). Central claims: relative Strategy Alignment gains above 50% for Qwen 3-4B Instruct (0.30 to 0.47/0.48) and Gemma-4-E4B (0.32 to 0.50), but only 4.54%/3.03% for MediPhi-Instruct (Table 2); heterogeneous gains across communities and coping types (meaning-focused posts best: Gemma +74.27%, Qwen +69.44%); and a post-training distribution shift toward problem-focused advice that suppresses emotion-focused strategies.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the COPES dataset?
COPES is a benchmark of 4,455 Reddit posts seeking non-clinical peer support, drawn from 55 mental-health subreddits (2011-2021). Each post carries coping-strategy labels plus a preferred response synthesized from community comments and a rejected response written to exhibit one of 11 harmful support patterns.
Does fine-tuning make LLMs better at peer support?
On COPES' community-alignment metric, yes for general-purpose models: relative Strategy Alignment rose over 50% for Qwen 3-4B (0.30 to 0.47-0.48) and Gemma-4-E4B (0.32 to 0.50), but only about 3-5% for the medical model MediPhi, and gains varied by community and by the coping strategy sought.
What side effects of post-training does the paper find?
Models drifted toward problem-focused advice and used emotion-focused strategies less; SFT+DPO also reduced topical overlap with reference responses across all models, and Qwen got worse (non-significantly) on the emotion-and-tone metric.
Is COPES publicly available?
The authors commit to releasing the dataset as Reddit post IDs rather than raw text, for privacy and platform-policy reasons; no code release is stated in the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extracted text is truncated in Sec. 2.3, Sec. 5.2, parts of the references, and appendix prompt tables (e.g., Table 12); some procedural details (e.g., DPO pair counts) are not visible.
- Figures 1-6 are parsed only from embedded captions and axis values in the text; visual distributions could not be independently verified.
- arXiv preprint (2609.21075); peer-review venue and status are unknown at review time.
- The dataset is promised via Reddit post IDs only; the release itself was not verifiable at review time, which tempers reproducibility judgments.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.