OBJECTION! Lawyer Agents Mitigate Guilty Bias in Legal Judgment Prediction
cs.CL · cs.AI
Quick verdict
Inference-time adversarial lawyer agent cuts false-guilty rate 82.93%→16.69% on real acquittals; compelling idea, but gains vary sharply by backbone.
Summary
The paper identifies 'guilty bias' in Legal Judgment Prediction (LJP): models trained on prosecutorial fact narratives over-predict guilty verdicts. It proposes OBJECTION, a training-free inference-time pipeline in which a single LLM plays both Judge and adversarial Lawyer Agent; within each stage of a trichotomous reasoning chain (offense, unlawfulness, culpability), the Judge issues an initial judgment, the Lawyer constructs a defense argument grounded in extracted facts, and the Judge re-decides, with early exit upon acquittal. The authors also release 'Natural Innocent', 3,412 real South Korean first-instance judgments (37.8% guilty) labeled by acquittal ground, addressing shortcut-prone synthetic innocence in prior benchmarks. On this dataset with Qwen-2.5-7B, OBJECTION lowers False Guilty Rate from 82.93% (fine-tuned SOTA LJPIV) to 16.69% with the best Macro-F1 (0.76). Ablations show generic critics fail (FGR 56.40%) while the role-constrained lawyer reduces FGR; removing structural anchors causes over-acquittal. Exact McNemar tests confirm FGR reductions in 20/27 settings, though OBJECTION significantly loses to Debate-Feedback in 3 of 9 — notably on Llama-3.1, where its Natural Innocent FGR (46.39%) exceeds even plain prompting (36.73%). Commercial-API results are mixed on real acquittals (GPT-4.1-mini FGR 58.46% vs Debate 54.92%). Limitations include keyword-based dataset annotation verified by a single investigator, a human evaluation panel drawn from law enforcement, and civil-law-only scope.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LJP models inherit guilty bias from prosecutorial fact narratives, and prior fixes — structured reasoning or training on synthetic innocent cases — fail to mitigate bias at inference time (Secs. 1–2; LJPIV FGR 82.93% on Natural Innocent).
- Injecting a role-constrained adversarial Lawyer Agent at each of the three trichotomous reasoning stages, training-free, cuts False Guilty Rate from 82.93% to 16.69% on real acquittals with the Qwen-2.5-7B backbone (Table 2).
- The effect stems from role asymmetry rather than critique per se: a generic Normal Critic yields FGR 56.40% vs 23.20% for the lawyer (Table 3, ELAM/Qwen).
- Synthetic innocence benchmarks induce shortcut overfitting, motivating the real-world Natural Innocent dataset of 3,412 Korean first-instance judgments (Appendix B).
Strengths
Normative metric framing tied to legal doctrine
FGR (FP/(TN+FP)) and FNR are defined to directly encode the presumption of innocence (Sec. 4.1), and all analysis is organized around them rather than accuracy alone; Table 15 further breaks FGR down by acquittal ground, showing the gain holds across offense (85.79→17.08), unlawfulness (86.31→17.40) and culpability (72.57→12.83).
Real-world acquittal benchmark with documented construction
Natural Innocent (3,412 Korean first-instance judgments; Appendix B) includes keyword tables (Table 7), verdict-type distribution (Table 9: 37.8% guilty, 42.9% offense-not-established, 12.6% justification, 6.6% no culpability), top charges (Table 10) and token statistics (Table 11). It exposes genuine SOTA failure: LJPIV scores FGR 82.93%, Macro-F1 0.32 (Table 2), confirming synthetic-innocence overfitting.
Ablations that isolate role asymmetry, not just 'adding an agent'
Above-average statistical and failure-mode transparency
Weaknesses
Backbone-dependent gains contradict the 'consistent' narrative
Overclaiming in the conclusion
Selective interpretation of commercial-API results
Dataset label integrity rests on a single annotator
Human evaluation panel and compute asymmetry
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets **guilty bias** in Legal Judgment Prediction (LJP): because LJP corpora consist of the fact sections of written judgments — prosecutorial narratives, in the authors' framing — models over-convict. It contributes (1) **OBJECTION** (Sec. 3), a training-free inference-time pipeline injecting a role-constrained adversarial Lawyer Agent into each stage of the trichotomous chain (offense → unlawfulness → culpability); (2) the **Natural Innocent** dataset of 3,412 real South Korean first-instance judgments with acquittal-ground labels (Appendix B); (3) evidence that structured reasoning and synthetic-innocence training leave bias intact at inference. Headline: FGR drops from 82.93% (fine-tuned LJPIV) to 16.69% on Natural Innocent with Qwen-2.5-7B.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'Guilty Bias' in Legal Judgment Prediction?
It is the tendency of LJP models — typically trained on the fact sections of written judgments, which are drafted from the prosecution's perspective — to over-predict guilty verdicts. The paper measures it as the False Guilty Rate (FGR): the share of truly innocent cases predicted guilty. On the paper's new real-acquittal dataset, a fine-tuned SOTA model misjudges about 83% of actually innocent defendants as guilty.
How does the OBJECTION framework work?
OBJECTION is a training-free, inference-time pipeline in which one LLM plays two prompted roles: a Judge and an adversarial Defense Lawyer. Judgment proceeds through three criminal-law stages (offense, unlawfulness, culpability). At each stage the Judge issues an initial judgment, the Lawyer writes a defense argument grounded in the case facts, and the Judge re-decides with that argument in context. If a stage negates guilt, the pipeline exits early with an acquittal and a granular acquittal label.
What is the 'Natural Innocent' dataset?
A set of 3,412 real first-instance criminal judgments from South Korean courts, with 37.8% guilty and the rest acquittals labeled by legal ground (42.9% offense not established, 12.6% justification such as self-defense, 6.6% no culpability). It was built by keyword filtering of public court records with police-investigator verification; a revised 3,200-case edition is released on Hugging Face.
How large is the improvement, and does it always hold?
With Qwen-2.5-7B on Natural Innocent, the False Guilty Rate falls from 82.93% (fine-tuned LJPIV baseline) to 16.69%, with the best Macro-F1 (0.76) in that setting. The improvement does not hold uniformly across backbones: with Llama-3.1, OBJECTION's FGR on Natural Innocent (46.39%) is worse than plain prompting (36.73%) and much worse than a debate baseline (7.69%), and exact McNemar tests show significant losses to that baseline on all three datasets with Llama-3.1.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–8 (including the case-study screenshots in Figs. 5–8 and the distribution bars in Fig. 3) are present only as captions in the extracted text; the plotted/graphical content was not extractable, so figure-internal values could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.