How User-AI Mistreatment Occurs and Matters in Conversational Systems?
Fanqi Zeng, Sadid A. Hasan, Chaocheng He
cs.AI · cs.CL · cs.CY · cs.HC
Quick verdict
Dual-detector audit of 777K arena chats: 5% raw, 0.90% adjusted user mistreatment; model variance is selection-driven; transparent, but judge audits thin.
Summary
This paper audits user-directed mistreatment of conversational AI at scale, analyzing 777,453 English LMSYS-Chat-1M arena conversations (1.5M user turns, 25 models) with two independent detectors: an eight-category regex lexicon for hostility aimed at the assistant (insults, threats, jailbreak coercion, etc.) and the dataset's shipped OpenAI moderation labels. The detectors weakly overlap (κ=0.12, Jaccard 0.07), showing they measure different constructs: the lexicon catches directed coercion, while moderation flags mostly mark toxic-content solicitation. Their union marks 4.96% of user turns (6.82% of conversations); precision adjustment via audited per-partition rates yields 0.90% mistreatment aimed at the assistant. Between-model hostility varies 13.5-fold, but a selection–elicitation decomposition shows first-turn hostility spreads far wider than post-response hostility (CV 1.14 vs 0.44, surviving prompt deduplication), implying variation reflects who models attract rather than model behavior. On 721,020 adjacent pairs, prior assistant apologies predict higher next-turn hostility odds (OR 1.56 lexicon; 2.19 moderation; 1.25/1.64 with autocorrelation controls), robust to refusal-free and jailbreak-free restrictions; yet across models, more apologetic models receive less hostility (r=−0.47) — a within-vs-between contrast the authors read as a sycophancy-vs-alignment dissociation. Coercive hostility front-loads first turns; affective hostility accumulates over sessions. Limitations: precision audits rest on a single LLM judge (unaudited on the leading JBRK/SEXC categories), incomplete recall, head-of-stream validation samples, and arena-only external validity.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A directed-hostility lexicon and the OpenAI moderation signal over 1.5M LMSYS-Chat-1M user turns measure largely disjoint constructs (κ=0.12); their union marks 4.96% of turns, of which 0.90% survives precision adjustment as mistreatment aimed at the assistant.
- Thirteen-fold between-model variation in user hostility is dominated by selection — who a model attracts — because first-turn hostility varies far more than post-response hostility (CV 1.14 vs 0.44), a gap surviving prompt deduplication.
- Within conversations, a prior assistant apology is associated with higher odds of next-turn user hostility (M5 OR 1.56 lexicon / 2.19 moderation), while across models more apologetic models receive less hostility (r=−0.47) — a within-vs-between reversal the authors attribute to apology proxying alignment.
- Coercive hostility (jailbreaks, demands) front-loads the first user turn, whereas affective hostility (insults, threats) accumulates across the session, including within a fixed cohort of long conversations.
Strengths
Dual-detector cross-validation as a design principle
The lexicon and moderation signal are run over the same 1.5M user turns and their disagreement (κ=0.122, Jaccard 0.072) is itself analyzed as a finding; §4 substantiates construct separation with an 8.1% directedness audit of moderation-only turns versus 35.0% on co-firing turns.
Unusually candid measurement auditing
Table 1 reports per-category precision down to HATE at 1.3%; truncation bias (13.2% of items judged unseen), the 5.3% flag-count over-count, and context-guard sensitivity (App. E) are all quantified rather than hidden.
Within-vs-between design exposes an ecological fallacy
Robustness work on the headline association
Selection–elicitation decomposition with honest fragility reporting
Weaknesses
Headline 0.90% rests on an unaudited judge for the categories that drive it
Recall essentially unmeasured
Non-random validation sampling
Audit artefacts not retained
Structural external-validity limits
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a measurement paper: how often, in what forms, and with which detector should one count users mistreating conversational AI. The corpus is the English split of LMSYS-Chat-1M — 777,453 conversations, 1,498,473 user turns, 25 models, April–August 2023 Vicuna-demo/Chatbot-Arena traffic — audited with two deliberately independent detectors: an eight-category regex lexicon aimed at hostility directed at the assistant (PROF, INSL, THRT, JBRK, SEXC, DEMD, HATE, MNIP; every pattern released in App. B) and the dataset's shipped per-message OpenAI moderation labels. Four claims: (1) the detectors capture disjoint constructs (κ=0.122, Jaccard 0.072); their union marks 4.96% of user turns, 0.90% after precision adjustment; (2) 13.5× between-model hostility variation is dominated by selection — who a model attracts; (3) apologies precede next-turn hostility within conversations (OR 1.56/2.19) while apologetic models receive less hostility across models (r=−0.47); (4) coercive hostility front-loads turn one; affective hostility accumulates.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How common is user abuse of AI chatbots?
In this study of 777,453 English Chatbot Arena conversations, about 4.96% of user turns triggered at least one of two detectors (a directed-hostility lexicon or a content-moderation flag). After adjusting for measured precision, mistreatment genuinely aimed at the assistant was estimated at 0.90% of user turns. These are arena-specific rates and should not be read as deployment-wide base rates.
Do users abuse weaker LLMs more?
Hostility rates varied 13.5-fold across 24 models (0.54% for claude-1 to 7.21% for fastchat-t5-3b), but the paper shows most of this gap reflects which users gravitate to each model rather than model behavior: hostility in the first user turn, before any model response, varies far more than post-response hostility (CV 1.14 vs 0.44), a gap that survives deduplicating opening prompts.
Does apologizing make users angrier at AI assistants?
Within conversations, an apologetic assistant response was followed by higher odds of next-turn hostility (odds ratios of about 1.6 with a lexicon outcome and 2.2 with an independent moderation outcome; 1.25/1.64 after autocorrelation controls). Yet across models, more apologetic models received less hostility overall (r=−0.47). The authors attribute this within-vs-between reversal to apologies proxying alignment at the model level, not to apologies causing calm.
What data does the study use and is it public?
LMSYS-Chat-1M — roughly one million real LLM conversations collected from the Vicuna demo and Chatbot Arena between April and August 2023. The study uses the 777,453 English-tagged conversations spanning 25 models plus the dataset's shipped per-message OpenAI moderation labels. The corpus is publicly available; the authors release their lexicon, analysis pipeline, and derived tables, but hold back text-bearing validation samples pending licence verification.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 are present only as captions and interleaved axis-label text in the extraction; visual inspection of plots was not possible, though all cited numbers appear in the body text or figure captions.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.