Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms
Rong Wang, Kun Sun, Yadong Guo
cs.CL
Quick verdict
Claims LLMs judge politeness more like each other than humans, showing neutral compression; largely convincing, multi-reference evaluation, though inferential rigor and label provenance are thin.
Summary
The paper evaluates whether LLM politeness judgments align with human pragmatic norms, using two English datasets: the Stanford Politeness Corpus (3,000 requests with continuous crowd scores) and an undocumented Hugging Face three-way corpus (3,000 balanced instances). Seven models (GPT-4.1, Claude 3.5 Sonnet, Gemini 2.5 Flash, Grok 3, DeepSeek, two Llama-3 variants) were queried via OpenRouter under six zero-shot prompt framings, with rule-based detection of 15 politeness strategies and a 318-instance expert audit (five linguistically trained L2 annotators, majority-vote consensus). Key results: mean inter-model correlation (r = 0.724) exceeds mean model-human correlation (r = 0.385) in every prompt condition; human inter-annotator reliability is itself moderate (pairwise r = 0.425, leave-one-out r = 0.562). Models align better on explicit surface cues and diverge on rapport-building and indirect strategies. Categorical predictions show systematic neutral compression: Neutral overproduction (NCI difference +1.0 to +41.2 pp) and Impolite underprediction across all model-prompt cells. On the audit subset, LLM-expert agreement (κ = 0.395) exceeds LLM-crowd agreement (κ = −0.252), yet Neutral overproduction persists versus both references. Limitations: undated model versions, English-only data, missing context, diagnostic-only audit sampling, and unknown provenance of the categorical reference labels.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLMs agree more strongly with one another (mean pairwise r = 0.724; Fleiss' κ 0.518–0.709) than with human references (model-human r = 0.385; Cohen's κ 0.24–0.35) across all prompt conditions.
- Model predictions exhibit systematic neutral compression — Neutral overproduction (NCI +1.0 to +41.2 pp) and Impolite underprediction (ISR +7.4 to +25.1 pp) in every model-prompt cell — persisting under expert consensus as reference.
- Model-human alignment is associated with explicit surface linguistic cues, while rapport-building and mitigation strategies occur more frequently in misaligned cases.
- On the 318-case expert-audit subset, models agree more with expert consensus (κ = 0.395) than with crowd labels (κ = −0.252), showing estimated alignment depends on the human reference chosen.
Strengths
Multi-reference design with a genuine human baseline
Section 4.4/Appendix C estimate leave-one-annotator-out human-consensus agreement (r = 0.562) against the best LLM (r = 0.433), and report human inter-annotator stats (pairwise r = 0.425, ICC A,k = 0.786, within-utterance variance 57.6%), so model-human disagreement is interpreted against realistic human variability rather than a fictitious gold standard.
Directional disagreement diagnostics beyond aggregate kappa
The paper defines NCI/NCIratio, ISR and ENSR (Section 4.3) and shows Neutral overproduction (+1.0 to +41.2 pp, mean +21.7 pp) and Impolite suppression (+7.4 to +25.1 pp) are uniformly positive across all 20 model-prompt cells, establishing neutral compression as a distributional property rather than a subset artifact (Section 5.2).
Expert audit cleanly separates reference ambiguity from model bias
Validated and transparent methodology
Disciplined causal hedging
Weaknesses
Expert-audit headline rests on a selection artifact
Dataset 2 reference labels have unknown provenance
Central comparative claim lacks inferential support
Non-reproducible inference protocol and dated models
Possible scoring-scale inconsistency in prompts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This EMNLP 2026 submission asks whether LLM *judgments* of politeness — not polite output — align with human pragmatic norms. Using 3,000 Stanford Politeness Corpus (SPC) requests with continuous crowd scores (Dataset 1) and 3,000 balanced instances from the undocumented `frfede/politeness-corpus` (Dataset 2), the authors run seven models (GPT-4.1, Claude 3.5 Sonnet, Gemini 2.5 Flash, Grok 3, DeepSeek, Llama-3/3.1-8B) through six zero-shot prompt framings (four for the categorical task), then layer on a rule-based detector for 15 politeness strategies and a 318-case expert audit by five linguistically trained L2 annotators. The four central claims — models agree more with each other (inter-model r = 0.724) than with humans (r = 0.385); alignment tracks explicit surface cues; categorical predictions show systematic neutral compression; and the estimated alignment depends on which human reference is used — are each supported by concrete numbers, and the release of prompts, outputs and scripts (github.com/rong4ivy/llm-politeness) is a real asset.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'neutral compression' in this paper?
It is the paper's name for the tendency of LLMs to over-predict the label 'Neutral' and under-predict 'Impolite' when classifying politeness. Across all model-prompt combinations tested, models produced Neutral 1 to 41 percentage points more often than the reference, and predicted Impolite 7 to 25 points less often — for example, only 5% of model predictions were Impolite versus about 25-31% of human labels on the audited subset.
Do LLMs agree with each other more than with humans about politeness?
Yes, according to this study. On continuous scoring, the average correlation between pairs of models was r = 0.724, while the average correlation between a model and human scores was only r = 0.385, and the inter-model advantage held in every one of the six prompt conditions. The caveat is that this gap was not formally significance-tested.
Are the datasets and code available?
The paper states a full release of prompts, metrics, model outputs and analysis scripts at github.com/rong4ivy/llm-politeness. Dataset 1 is the public Stanford Politeness Corpus; Dataset 2 is the public frfede/politeness-corpus from Hugging Face, though its label-construction procedure is undocumented, which the authors acknowledge.
Does this mean LLMs cannot detect impoliteness at all?
No. The models detect overtly rude language well (explicit cues like direct commands and negative words align closely with human judgments) but systematically hedge: they collapse socially marked judgments — both strongly polite and impolite — toward Neutral, and especially avoid the Impolite label when rudeness is indirect or context-dependent.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Middle sections of the extracted text are truncated (parts of Section 4.3, 5.3, 6.1 and the end of Appendix B), so some prose was reconstructed from tables and surrounding context.
- Figures 1-5 are unavailable for analysis from text; all quantitative judgments rely on in-text numbers and tables.
- Tables 4, 10 and the P3/P6 continuous templates appear partially truncated; their full contents were not verifiable.
- Facts such as conference venue (EMNLP 2026) and the GitHub release were taken from the text as stated and not externally verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.