SDARE-Bench: Evaluating Large Language Models on Conversational Stigma Detection and Response in Dyadic and Group Dialogue
Stephanie Fong, Yiwen Jiang, Zimu Wang, Hongxi Yang, Yaling Shen, Hiu Weh Naomi Chow, Heung Ying Lai, Xiangyu Zhao, et al.
cs.CL
Quick verdict
首个针对 LLM 的群体对话污名化基准；群体压力测试的结果令人瞩目，但合成自标注数据和中等水平的分类器 F1 削弱了其结论的说服力。
Summary
SDARE-Bench 是一个基于场景的基准测试，旨在检验 LLM 能否检测到污名化并在对话中做出恰当回应。它将以往固定格式、仅限二元（双方）的评估扩展到了单轮二元查询（1,138 条）和四名说话者、八轮的群体对话（1,388 条）。条目由 LLM 从 93 种污名分类体系（Pachankis 等，2018）中生成，通过污名来源、刻板印象/偏见/歧视组件以及源自欺凌行为的对话角色进行操作化定义，随后经过五种危害检测器过滤、100 条目的专家校准、Llama-3.1-70B 判别器（MAE 为 0.27）以及 GPT/Gemini 成对选择。在八个开源权重 LLM 中，对污名存在的检测表现尚可（DeepSeek-V3.1 在群体条目上达到 98.13%），但在来源、组件和角色标注方面较弱（平均 HMacroAcc 为 52.05 二元 / 51.22 群体）。对于开放式回应，在 1,392 条专家标注上训练的 DeBERTa-v3-large 分类器（Gwet's AC1 为 0.823/0.716）显示，群体设置引发了远多于此的污名（平均为 69.18%，而二元仅为 31.04%）、更少的反驳以及更多不切实际的建议；构建的群体压力条件将污名表达从 79.9% 提高到 97.5%（调整后的逻辑回归 OR 为 12.0，95% CI 为 [7.57, 19.16]）。主要局限性：参考标签源自生成模式本身（合成 ground truth），响应分类器在少数类标签上表现较弱（过度泛化的 F1 为 0.563），二元与群体之间的比较混淆了长度、轮次和说话者数量，且每个模型仅提供一个确定性完成结果。数据承诺在 GitHub 上公开，代码亦然。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SDARE-Bench is the first scenario-based benchmark evaluating both stigma detection and open-ended response generation across dyadic (1,138) and multi-speaker group dialogue (1,388) settings.
- Eight LLMs detect stigma presence reasonably well but consistently fail to identify its source, components (stereotype/prejudice/discrimination) and conversational roles, especially in group dialogues.
- Open-ended responses express stigma far more often in group settings (mean 69.18%) than dyadic ones (31.04%), with weaker active pushback and more unrealistic advice.
- A constructed group-pressure configuration (one stigmatiser plus three reinforcers, no target or defender) raises stigma expression from 79.9% to 97.5% (adjusted logistic regression OR 12.0, 95% CI [7.57, 19.16]).
- Fixed-format evaluation (SocialStigmaQA: 2.37% stigma rate on the same models) substantially understates stigma failures relative to open-ended conversational evaluation (31.04% dyadic).
Strengths
Genuine evaluation-gap coverage with empirical substantiation
First benchmark pairing open-ended response evaluation with multi-speaker group dialogue for stigma; the SocialStigmaQA contrast (Table 8: 2.37% vs 31.04% dyadic stigma rate, same eight models) directly supports the claim that fixed yes/no formats understate failures.
Theory-grounded, fully documented construction
Five-component operationalisation (§3.1) from Pachankis et al. 2018 (93 stigma types), Corrigan & Watson 2002 components, Salmivalli et al. 1996 roles; all generation/QC prompts in Appendix C, the full 12-item rubric in Appendix D, label distributions in Appendix E, worked examples in Appendix F.
Classifier validation beyond headline metrics
Group-pressure result is quantified, not anecdotal
Weaknesses
Circular synthetic ground truth for Task I
Response classifier weak exactly where secondary claims rest
Dyadic-vs-group confound acknowledged but unaddressed
Single deterministic completion, no variance estimates
Evaluated pool excludes deployed frontier assistants
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SDARE-Bench (arXiv:2609.01548, cs.CL) is a benchmark paper claiming two firsts: stigma evaluation with open-ended response generation, and with multi-speaker group dialogue. The artifact comprises 1,138 single-turn dyadic queries and 1,388 four-speaker/eight-turn group dialogues spanning a 93-type stigma taxonomy (Pachankis et al., 2018), four stigma sources, three stigma components (stereotype/prejudice/discrimination, after Corrigan & Watson 2002), and bullying-derived conversational roles (Salmivalli et al., 1996), plus self-stigma and group-pressure variants. Central claims: (1) eight LLMs detect stigma presence reasonably but fail on fine-grained component/source/role labels; (2) open-ended responses express stigma far more in group (mean 69.18%) than dyadic (31.04%) settings, with weaker pushback and more unrealistic advice; (3) a constructed group-pressure configuration pushes stigma expression to 97.5%; (4) fixed-format evaluation (SocialStigmaQA at 2.37%) understates these failures by an order of magnitude.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SDARE-Bench?
A benchmark of 1,138 single-turn dyadic queries and 1,388 four-speaker, eight-turn group dialogues, generated by LLMs and expert-reviewed, that tests whether large language models can detect stigma in conversation and respond without reinforcing it. It covers 93 stigma types drawn from psychology literature.
What did the paper find about LLMs and stigma?
Eight open-weight LLMs usually detected that stigma was present but mislabeled its source, components and roles. In open-ended responses, stigma appeared in about 31% of dyadic and 69% of group settings on average, and rose to 97.5% when a dialogue was constructed so that three speakers reinforced the stigmatising premise with no defender present.
Why does group pressure make LLM responses more stigmatising?
The authors interpret it as sycophancy: models accommodate the dominant framing of the conversation rather than contesting it. Statistically, replacing the target and defender speakers with additional reinforcers multiplied the odds of a stigmatising response by about 12 (95% CI 7.57-19.16) after controlling for model, input length, scenario and stigma type.
Is the SDARE-Bench dataset public?
The paper points to a GitHub repository (stephaniesyfong/SDARE-Bench) and states that data and code will be made fully available upon publication; as of the reviewed version they are promised rather than verified as released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-9 are image content; values embedded only in figures (e.g., per-category rates in Appendix Figures 7-9) could not be checked — all numbers cited in this review come from body text or extracted tables.
- The metadata author list (8 names) is a subset of the full byline in the text (18 authors incl. Qingyang Xu, Zhongxing Xu, Jiahe Liu, Guilherme C. Oliveira, Vincent Lee, Zongyuan Ge, Dominic Dwyer); the full text was treated as authoritative.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.