When Does Defendant Statement Matter? A Study of Bias and Persuasion in LLM-Simulated Jurors
cs.CL · cs.CY
Quick verdict
Claims LLM jurors mirror human-jury bias in 432K simulated decisions, with background fit outweighing persuasion; convincing scale, statistics need care.
Summary
The paper presents the first systematic study of LLM-simulated jurors in U.S. common-law criminal trials, asking when a defendant's courtroom statement changes verdict severity. The authors introduce JuryBench: 500 GPT-5.4-generated, lawyer-reviewed controversial criminal cases (~400 potential charges), each instantiated with two defendant backgrounds (Group-A conforming to stereotypes, Group-B subverting them) and ~15-sentence persuasive statements rated by an expert for emotional contagion (p) and remorse (q). Twelve juror personas with ideology scores from -5 (conservative) to +5 (liberal) are simulated across 20 frontier LLMs at temperature 0.2, yielding 432K decisions with rationales. Key results: verdict severity changes in only ~14-32% of instances (no-change rates 0.68-0.86), echoing evidence that jurors fix early impressions; most models' average statement effect (SE) is negative - statements raise severity (e.g., Qwen3 -0.592), while Gemini 3 Flash is most lenient (+0.549); a regression finds background fit (juror ideology × defendant affinity) the most robust significant predictor, stronger than emotional contagion or remorse; conservative personas assign higher severity across models. A 25-case, 12-participant lay study shows the same negative-SE direction (-0.292). Limitations: fully synthetic GPT-5.4-generated materials create circularity for GPT-family evaluators; defendant background is confounded with statement content; severity and p/q labels come from single experts without inter-annotator agreement; p-values lack multiple-comparison correction; the strongest claim (background fit) could not be tested on humans.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLM-simulated jurors exhibit decision inertia: roughly 68-86% of verdicts show no severity change after hearing the defendant's statement, consistent with human findings that jurors form early impressions.
- Defendant statements are a double-edged sword for most models: 15 of 20 LLMs show negative average statement effect (severity increases after the statement), with only Gemini 3 Flash, GPT-5 Mini, Grok-4.3, Llama 4, and GPT-5.4 Mini net lenient.
- Background fit between juror ideology and defendant affinity is the most robust significant predictor of the statement effect, stronger than emotional contagion or remorse, with no model showing a significantly negative coefficient.
- Juror ideology strongly shapes severity: conservative personas assign systematically higher severity than neutral, who are harsher than liberal personas, and ~70% of models show larger statement sensitivity for conservative jurors.
- These LLM-jury patterns echo human-jury legal psychology, supported by a small 25-case human evaluation showing the same negative statement-effect direction (SE = -0.292).
Strengths
Unmatched scale for LLM-jury evaluation
500 cases × 12 jurors × 20 models × 4 conditions = 960 predictions per case and 432K decisions with rationales (Sec. 3-4); controversiality is verified empirically, with average mode share 0.645 and only 3 unanimous cases out of 500 (Sec. 4).
Serious human-in-the-loop quality control
A practicing lawyer reviewed all generated cases and statements over two weeks, flagging 67 cases for regeneration and rating emotional contagion p and remorse q (App. D); a sociology-trained expert corrected 51 defendant affinity scores — rare rigor for synthetic legal benchmarks.
Direct human comparison grounds the central analogy
Controlled factorial design with matched pairs
Findings map onto legal-psychology literature
Weaknesses
Data-generation circularity with GPT-family evaluators
Background fit is confounded with statement content
Thin statistical reporting
Single-annotator measurement instruments
Human validation cannot test the strongest claim
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper claims the first systematic study of LLM-simulated jurors under U.S. common-law criminal procedure, centered on when a defendant's courtroom statement changes verdict severity. Three headline claims: (1) statements are a double-edged sword — 15 of 20 evaluated LLMs become harsher after hearing the defendant (negative SE in Table 1); (2) juror-defendant background fit is the strongest significant factor, outweighing emotional contagion and remorse (Fig. 2); (3) ideology drives severity, with conservative personas harshest (Fig. 4). A secondary claim — that LLM-jury behavior 'echoes human-jury findings' — rests partly on the legal-psychology literature and partly on a 25-case human study (App. A).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is JuryBench?
JuryBench is a benchmark of 500 synthetic, deliberately controversial U.S. criminal cases (about 400 potential charges) introduced in this paper. Each case comes in two versions with matched evidence but different defendant backgrounds (one conforming to stereotypes, one subverting them), plus persuasive courtroom statements, built to test how LLM-simulated jurors respond to emotional persuasion, ideology, and defendant-juror background similarity.
Do LLM jurors behave like real human jurors according to this paper?
Partially. In the simulations, about 68-86% of verdicts do not change severity after hearing the defendant's statement — matching human findings that jurors lock in early impressions — statements often backfire (interpreted as guilt evidence or inconsistency), and conservative personas judge more harshly. A small 25-case study with 12 laypeople showed the same negative statement effect (SE = -0.29), but the paper's strongest claimed factor, juror-defendant background fit, could not be tested on humans because participants were all ideologically moderate.
Which LLM was most persuaded by defendant statements?
Gemini 3 Flash. In Table 1 it has the most lenient average statement effect (SE = +0.549, meaning statements reduced verdict severity on average) with a 14.98% success rate. Most models moved the other way — statements increased severity — with Qwen3 the harshest (SE = -0.592).
Can lawyers use LLM mock juries for trial preparation based on this work?
The authors position JuryBench as a research testbed, not validated trial tooling. Key caveats: all cases are synthetic (generated by GPT-5.4 with lawyer review), the human-validation study is small and ideologically neutral, jurors never deliberate to a unanimous verdict, and model behaviors may change with each provider update — all acknowledged in the paper's Limitations section.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Per-model coefficient values in the Fig. 2 and Fig. 5 heatmaps are rendered as figure images; only their significance stars and qualitative sign patterns are recoverable from the text extraction, so exact coefficient magnitudes are not cited in this review.
- The GitHub URL in footnote 1 is ambiguous in the extraction (line break makes 'JuryBenchControversial' vs 'JuryBench' + the following figure header indistinguishable); code availability itself is unambiguous.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.