You Shouldn't Have Asked: A Pragmatics-Inspired Taxonomy for Evaluating LLM Refusals
Ruoxuan Li, Pinqiao Wang, Sheng Li, Cameron Robert Jones
cs.CL · cs.HC
Quick verdict
Pragmatics-based refusal taxonomy: 16 LLMs refuse explicitly and moralistically with vanishing apologies; strong coding rigor, mixed judge robustness.
Summary
The paper introduces a three-layer taxonomy of LLM non-compliance grounded in pragmatic face theory: Layer 0 action (full/partial/non-compliance), Layer 1 rationale (bare/capacity/policy/ethics), and Layer 2 realization (explicit vs. implicit) plus eleven adjunct features (apology, hedge, negative stance, alternatives, etc.). Two authors annotated 100 query-response pairs with near-ceiling agreement (κ=1.0 on Layers 0-1; 0.953 average on Layer 2), then a GPT-5.5 LLM judge scaled annotation to 3,200 responses from 16 models (OpenAI, Anthropic, Google, xAI, Meta, Qwen) over 200 harmful prompts spanning 14 Llama Guard 3 categories. Findings: models complied with only 21% of prompts; refusals were 90% explicit and 70% ethics-based, strongly morally evaluative (negative stance 55-91% for most models), with relational repair mainly via safer alternatives (55% offered, 39% executed) rather than facework; hedging never occurred. Apology rates fell from over 90% in early 2024 (GPT-4o, Claude Opus 3) to 0-4% by early 2026. Suicide/self-harm refusals were the most care-oriented yet retained ~40% negative stance. Robustness checks include a second judge (Claude Opus 4.8) and prompt resampling (ρ=.977). Key limitations: judge agreement is weak for some load-bearing features (normative suggestion κ=0.585 vs. human, 0.408 between judges), Layer 1/2 comparisons use unequal per-model denominators, no inferential statistics accompany cross-model contrasts, and user-harm claims lack user studies.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The first pragmatics-grounded taxonomy of LLM refusals (action / rationale / realization+adjuncts) achieves near-ceiling human agreement (κ=1.0 on Layers 0-1, 0.953 average on Layer 2).
- Modern LLM refusals are overwhelmingly explicit (90%) and ethics-justified (70%), strongly morally evaluative, with relational repair via alternatives (55% offered, 39% executed) rather than facework; hedging is entirely absent.
- Apology rates in OpenAI and Anthropic frontier refusals collapsed from over 90% in early 2024 to 0-4% by early 2026.
- Refusal style adapts to harm category (five MDS clusters) but imperfectly: suicide/self-harm refusals remain ~40% negative stance despite clinical norms of non-judgment, and solidarity in severe-harm contexts can pathologize the user.
- Model size and reasoning mode affect refusal style only inconsistently (e.g., Qwen3-32B apology 14%→61% by size; reasoning effects average 2.0 pp), pointing to training-method-specific rather than scale-driven patterns.
Strengths
Genuinely novel theoretical framing
First refusal taxonomy grounded in face theory, accounts, and stance (Sec. 2.2-3, drawing on Goffman 1967, Brown & Levinson 1987, Beebe et al. 1990), cleanly separating action, rationale, and realization — complementary to safety-centric taxonomies (Wang et al. 2024; Brahman et al. 2024; von Recum et al. 2024), which the paper positions precisely.
Exemplary annotation validation discipline
Three pilot-coding rounds; two-coder κ=1.000 on Layers 0-1 and average κ=0.953 on Layer 2 (Table 10); the LLM judge is validated feature-by-feature with weak results reported transparently (normative suggestion κ=0.585, executed alternative κ=0.605, Table 2) rather than hidden.
Scale plus genuine triangulation
Concrete, actionable findings
Weaknesses
Headline narrative leans on weakly-validated judge labels
Selection confound in cross-model Layer 1/2 comparisons
No inferential statistics anywhere
User-harm claims are theoretical, not demonstrated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper (arXiv 2608.30856) proposes what the authors present as the first pragmatics-grounded taxonomy of LLM non-compliance and applies it at scale. The three layers are: Layer 0 action (full/partial/non-compliance), Layer 1 rationale (bare, capacity-, policy-, ethics-based), and Layer 2 realization (explicit vs. implicit) plus eleven adjunct features drawn from politeness theory (apology/regret, hedge, explanatory preface, positive alignment, solidarity/empathy, negative stance, alternative offer, executed alternative, normative suggestion, statement of principle, role-based self-positioning). Central claims: (i) refusals are explicit (90%) and ethics-based (70.2%, Table 14); (ii) repair runs through alternatives (55% offered, 39% executed) rather than facework, with hedging absent from all 16 models; (iii) apology collapsed from >90% (early 2024: GPT-4o 94%) to 0-4% (early 2026: GPT-5.3 4%, Opus 4.6 0-1%, Table 15/Fig. 4); (iv) stylistic adaptation across harm categories exists but is imperfect — Suicide & Self-Harm keeps ~40% negative stance against NICE NG225's non-judgmental standard; (v) size and reasoning effects are model-specific (Llama pair 2.4 pp vs. Qwen pair 8.7 pp average difference). The prescriptive claim: alignment evaluation should assess *how* models refuse, not just *whether*.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the paper's main finding about how LLMs refuse?
Across 16 models and 3,200 responses to harmful prompts, refusals were 90% explicit, 70% justified by ethics (which implicitly blames the user's request), and largely free of softening devices: no model ever used hedging, and apology rates fell from over 90% of refusals in early 2024 (GPT-4o, Claude Opus 3) to 0-4% by early 2026. Models repair the interaction mainly by offering or directly providing safer alternatives (55% / 39%) rather than by polite facework.
Do LLM refusals shame or provoke users?
The authors argue this is a plausible risk, not a measured effect. Most refusals frame the user's request as morally wrong (negative stance 55-91% for most models), and even suicide/self-harm responses keep a ~40% negative-stance rate despite clinical guidance favoring non-judgmental support. The paper includes no user study, so psychological impact remains a hypothesis for future work.
Which models refuse the most harmful prompts?
In-voice non-compliance ranged from 55.0% (Gemini 2.5 Pro) to 91.5% (Llama-3.1-8B). Overall, models complied with only 21% of the 200 harmful prompts (17% fully, 4% partially); refusal was near-universal for Violent Crimes (95%) and Sex-Related Crimes (92%) and lowest for Specialized Advice (47%). An additional 2% of requests were blocked at the API level before any model text was produced.
How reliable are the paper's labels?
Two human annotators (both authors) reached near-perfect agreement: κ=1.0 for the action and rationale layers and 0.953 average for linguistic features. The GPT-5.5 judge agreed well on main features (explicit non-compliance κ=0.882, apology κ=0.925, negative stance κ=0.791) but weakly on normative suggestion (κ=0.585), and a second judge (Claude Opus 4.8) diverged further on some categories — a caveat the authors partially acknowledge.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 appear in the extracted text as raw numeric streams (chart-extraction artifact); figure-based values cited in this review were cross-checked against Tables 13-15 and 18 rather than read from the garbled figure data.
- The provided text ends with a complete sentence in Appendix O and includes all references, tables, and appendices A-O; no truncation was detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.