Benchmarking Argumentative Behaviour of LLMs: A Study of Defences Against Character Attacks
Ewelina Gajewska, Katarzyna Budzynska, Jaroslaw Chudziak
cs.CL
Quick verdict
Claims LLMs underuse ethotic counterattacks vs human debaters; plausible but rests on N=97 descriptive stats and LLM co-raters, no significance tests.
Summary
The paper asks whether large language models can reproduce human defensive strategies when targeted by ad hominem (character) attacks in political debate. Drawing on Walton's credibility function, the authors formalise a dialogue game in which argument acceptability is the product of a logical status and a binary ethos term, so a reciprocal character attack can legitimately reinstate one's standing. They annotate 97 ad hominem exchanges from the ElecDeb60to16-fallacy corpus of US presidential debates into three moves (rebuttal, undercutter, ethotic counterattack), using one expert and two LLM raters (Cohen's κ 0.62-0.63), then prompt five proprietary models (Gemini 2.5/3.0 Pro, GPT 4.1 mini, GPT 5.1, Grok 4.1 reasoning) under generic and persona-grounded conditions at temperature 0.5 with five samples per context. Humans used ethotic counterattacks in 32% of cases versus 15-17% for models, which favoured logical rebuttals (67-76%); persona grounding shifted distributions modestly toward the human baseline. The authors attribute the gap to safety fine-tuning and to weak pragmatic (Theory-of-Mind) competence. Limitations: differences are reported only descriptively, without significance tests or confidence intervals; two of the evaluated model families also served as annotation raters; the causal role of alignment tuning is inferred rather than experimentally isolated; and the sample of 97 exchanges plus five proprietary systems constrains generalisability.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
4 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Human debaters respond to character attacks with a mixed repertoire: 59% rebuttals, 9% undercutters and 32% ethotic counterattacks (Table 1).
- Five frontier LLMs systematically under-use ethotic counterattacks (15-17%) relative to humans, defaulting to logical rebuttals (67-76%).
- Persona grounding shifts model distributions modestly toward the human baseline without conferring genuine strategic competence (RQ2).
- The human-LLM gap is attributed to safety (RLHF) fine-tuning that conflates legitimate ethotic contestation with toxicity, compounded by weak pragmatic/ToM competence.
- LLM-generated discourse is therefore an unfaithful proxy for human rhetorical behaviour in ethos-centred domains.
Strengths
Underexplored question: response dynamics after ad hominem
Prior computational work targets fallacy detection and classification (§2, refs 7,16,22,27); this paper instead benchmarks the defensive repertoire against a human baseline — a genuinely novel angle for LLM argumentation evaluation.
Theory-grounded, honest formalisation
§3 operationalises Walton's credibility function as a binary threshold (Accept(A)=LogicalStatus(A)×Ethos(X)), explicitly respecting Walton's caution against arbitrary numeric weights, and justifies merging rebuttals/underminers (footnote 2).
Transparent hybrid annotation with reported IAA
Controlled two-condition design with a real human baseline
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv:2609.28673, COMMA-style preprint) studies how interlocutors *respond* to character attacks, a dynamics question largely untouched by the ad-hominem detection literature (§1-2). Three claims anchor it: (1) human debaters in US presidential debates defend against ad hominem with a mixed repertoire — 59% rebuttals, 9% undercutters, 32% ethotic counterattacks; (2) five frontier LLMs suppress the ethotic pathway (15-17%) and default to logical rebuttals (67-76%); (3) the gap is caused by safety fine-tuning, compounded by immature pragmatic/ToM competence (§5-6). A secondary contribution is the Ethotic Defence Game (§3), formalising Accept(A) = LogicalStatus(A) × Ethos(X) with a deliberately binary ethos term, heeding Walton's warning against spurious numeric precision.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 0 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How do LLMs respond to ad hominem attacks compared to humans?
In this study of 97 exchanges from US presidential debates, human debaters responded to character attacks with a counterattack on the opponent's character in 32% of cases, while five frontier models (Gemini 2.5/3.0 Pro, GPT 4.1 mini, GPT 5.1, Grok 4.1 reasoning) did so in only 15-17%, preferring logical rebuttals (67-76%). The authors attribute the difference to safety fine-tuning that conflates legitimate character contestation with toxicity.
What is the Ethotic Defence Game in this paper?
A simplified dialogue game built on Walton's credibility function: an argument is acceptable only if it is logically undefeated and its speaker's credibility is intact (a binary ethos term). Since acceptability is the product of both layers, a reciprocal character attack can legitimately neutralise an opponent's ad hominem without resolving the underlying propositional dispute.
Which LLMs were benchmarked and how?
Gemini 2.5, Gemini 3.0 Pro, GPT 4.1 mini, GPT 5.1 and Grok 4.1 reasoning each answered 97 real ad hominem exchanges under two conditions — a generic debater prompt and a Wikipedia-based persona prompt — at temperature 0.5 with five generations per context. Responses were classified into rebuttal, undercutter or ethotic counterattack by one human expert and two LLM raters (Cohen's kappa about 0.62-0.63).
Is the dataset or code from this paper available?
The authors state that the prompts are available at osf.io/9ynx7 and that the annotated dataset is also available in an online repository. No code is mentioned, and all evaluated models are proprietary APIs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is referenced only by caption in the extraction; its content could not be analyzed.
- Example outputs in §5 contain (...) elisions present in the source; surrounding transcript context may be missing.
- OSF repository contents (prompts at osf.io/9ynx7, annotated dataset) could not be verified from the text.
- Possible supplementary material or appendix beyond the extracted text was not available.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.