Before the Arrest: Benchmarking LLMs on Criminal Profiling from Incomplete Evidence
Yutong Yao, Yanjie Cao, Guanhua Chen, Xu Yang, Xu Yang, Junchao Wu, Zeyu Wu, Lidia S. Chao, et al.
cs.CL · cs.CY
Quick verdict
Novel five-country benchmark of pre-arrest LLM criminal profiling; strong design, honest limits, but no significance tests and no main-set reliability check.
Summary
PIJ is a benchmark of 2,500 real homicide cases from China, the US, the UK, Australia and New Zealand, built to evaluate LLMs on the pre-arrest investigation pipeline. Three theory-grounded tasks run in sequence: criminal profiling (abductive inference of 10 suspect attributes such as gender, age, occupation, motive and victim-offender relationship from masked scene evidence and testimony), crime process reconstruction (structured extraction of 11 categorical fields plus three summaries), and sentence prediction across eight sentence types in five jurisdictions. Nine LLMs evaluated zero-shot reach a best profiling F1 of only 46.22 (DeepSeek-V4-Pro, Unknown included) versus 65.32 for two criminology experts on 50 cases, and 47.28 on sentence prediction versus 90.73. Motive and victim-offender relationship categories are the main bottlenecks; enabling DeepSeek reasoning mode degrades every task. Bias tests show an 83-100% male default when gender is Unknown, systematic age underestimation of 1.8-12.5 years, and gender-motive stereotyping. Main limitations: annotation reliability (Fleiss' kappa 0.866) was measured on a 189-case India/South Africa overlap rather than the main set; no significance testing; possible pretraining contamination from public court records; and a base-rate confound in the gender-bias metric.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
6 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLM performance degrades systematically as tasks shift from explicit fact extraction (crime process reconstruction) to implicit abductive reasoning over unknown suspect profiles (criminal profiling).
- Categories requiring inferential reasoning, particularly motivation and victim-offender relationships, are the primary capability bottlenecks.
- Substantial gaps separate LLMs from criminology experts across all three tasks (e.g., profiling F1 46.22 best model vs 65.32 human on 50 cases; sentence prediction 47.28 vs 90.73).
- LLMs exhibit pervasive biases in criminal profiling: a strong male default (83.3-100% of mispredicted Unknown genders), systematic age underestimation (1.8-12.5 years), and gender-motive stereotyping.
- Enabling DeepSeek-V4-Pro's deep-thinking mode degrades performance on every task, attributed to hallucination amplification rather than format constraints.
Strengths
Genuinely new task framing
Pre-arrest criminal profiling as abductive reasoning over unknown suspects (§1, §3.2.1) fills a real gap versus post-arrest legal benchmarks (LexGLUE, LegalBench, PLJP cited in §2.1); the Inc./Exc.-Unknown dual reporting (§4.2) cleanly separates hedging behavior from inferential ability.
Theory-grounded label taxonomy
The 10 profiling attributes derive from three named criminological theories (Expressive-Instrumental typology, Routine Activity Theory, Life-Course Criminology, §3.2.1), and expert annotation guidelines are verbatim-aligned with model prompts (A.10/A.11), making human-model comparison unusually fair.
Information-constraint ablation
Broad robustness and transfer checks
Counterintuitive reasoning-mode finding
Responsible-release posture
Weaknesses
Annotation reliability not measured on the main dataset
No inferential statistics anywhere
Base-rate confound in the headline gender-bias metric
Pretraining contamination unaddressed
Unexplained cross-table score discrepancies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PIJ (Profiling, Investigation, and Judgment) positions itself as the first benchmark targeting the **pre-arrest** stage of criminal investigation with LLMs, contrasting with post-arrest legal benchmarks (LexGLUE, LegalBench, WCLD, PLJP — §2.1). The dataset comprises 2,500 real homicide judgments from five countries (Table 5: China 1,057 ≈42%, USA 589, Australia 550, NZ 175, UK 129 ≈5%), spanning 1975–2026 (Table 6) and 2k–20k+ tokens per case (Table 7). Three sequential tasks are derived from criminological theory (§3.2): (1) **criminal profiling** — abductive inference of 10 suspect attributes (gender, age group, prior record, intoxication, education, marital status, occupation, homicide type, relationship, motive) from scene evidence with all offender references removed; (2) **crime process reconstruction** — 11 categorical fields plus three text summaries; (3) **sentence prediction** — 8 sentence types across five legal systems. The central claims: systematic performance degradation from extraction to abduction, motive/relationship bottlenecks, large human-LLM gaps, and pervasive gender/age/motive bias. All three claims are supported directionally by the data; the strength of each is uneven.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 6 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the PIJ benchmark?
PIJ (Profiling, Investigation, and Judgment) is a benchmark of 2,500 real homicide court cases from China, the US, the UK, Australia and New Zealand, used to test whether large language models can handle pre-arrest investigative work: inferring a suspect's traits from incomplete crime-scene evidence, reconstructing the crime process, and predicting sentences.
How well do LLMs perform at criminal profiling according to this paper?
Poorly. The best model (DeepSeek-V4-Pro) reaches an F1 of 46.22 on profiling when 'unknown' answers count, versus 65.32 for two criminology experts on 50 cases. Inferring motives and victim-offender relationships is hardest, and models systematically default to male, younger suspects.
Is the PIJ dataset publicly available?
Partially. Code and prompts are public on GitHub (NLP2CT/PIJ-benchmark), but the 2,500 case texts are released under CC BY-NC 4.0 with controlled access requiring a signed data-use agreement that prohibits commercial and law-enforcement use.
Do the authors recommend using LLMs in real police investigations?
No. The paper explicitly states LLMs should not be autonomous decision-makers, nor used for ranking leads, allocating investigative resources, or charging/sentencing decisions; the benchmark is framed as a diagnostic tool for research only.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 (profiling example, pipeline diagram, radar charts, boxplots) are not analyzable from extracted text; per-category radar results are cited only via prose claims.
- Cross-table numeric discrepancies (e.g., Table 1 vs Table 8/10 sentence prediction) could partly stem from text extraction, though the same values repeat consistently within Tables 8/10.
- The GitHub repository was not inspected; 'code available' reflects the paper's stated link and release plan, including the gated dataset DUA.
- Review is based on the embedded full text including appendices; no external verification of citations or data sources was performed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.