How Much is a Human Right Worth? ECtHR-NPD: A Benchmark for Predicting Non-Pecuniary Damage Awards
Yanyi Pu, Damian A. Gonzalez-Salzberg, Zheng Yuan, Nikolaos Aletras
cs.CL
Quick verdict
First ECtHR non-pecuniary-damage award prediction benchmark (14.6k cases); rigorous leakage control, but best model beats a median constant by just 10.2%.
Summary
The paper introduces ECtHR-NPD, a benchmark of 14,575 European Court of Human Rights judgments labeled with case-level non-pecuniary damage (NPD) awards in nominal euros, framing discretionary monetary remedies as continuous regression. Inputs comprise case metadata, the FACTS section, violated articles, and state-year macroeconomic covariates; applicants' Article 41 claims, the Court's just-satisfaction reasoning, and operative provisions are withheld to prevent target leakage. Targets come from a multi-source extraction pipeline with eight validation checks and a legal zero-award taxonomy. Chronological 70/10/20 splits produce a 2,897-case test pool partitioned into quota-matched ID (1,000), OOD (1,897), and an overlapping Challenging view (699). Six method families are compared: constants, gradient-boosted trees, retrieval, fine-tuned encoders, prompted decoder LMs (Qwen3.5 variants, GPT-OSS-20B, GPT-5.4), and ReAct agents. CatBoost achieves the best MAE (9,881 EUR), only 10.2% below the train-median constant; prompted LMs and agents underperform, and paired bootstrap tests confirm only some gaps are significant. All families fail at zero-award recognition (retrieval's Zero-F1 tops out near 0.53) and high-award calibration, with MAE roughly doubling on the Challenging view. An expanded agent given claims and Article 41 reasoning halves MAE (8,810 EUR), showing the withheld material dominates the signal. Limitations include possible label noise from LLM-assisted extraction, nominal-euro targets, and single-run LLM evaluations.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ECtHR-NPD is, to the authors' knowledge, the first benchmark for predicting discretionary Article 41 non-pecuniary damage awards at the ECtHR as continuous regression, with 14,575 validated case-level euro targets.
- No sophisticated LM or agentic method consistently outperforms the strongest feature-based baseline; CatBoost's MAE of 9,881 EUR is only 10.2% below the training-median constant predictor.
- All model families fail to identify zero awards and to calibrate high awards, with MAE roughly doubling on the legally identified Challenging view.
- Applicants' claims and the Court's Article 41 reasoning are the dominant predictive signal: exposing them to ReAct agents nearly halves MAE (18,220 to 8,810 EUR), justifying the benchmark's strict input separation.
Strengths
Genuinely novel task for legal NLP
First benchmark (per authors) treating discretionary monetary remedies as continuous regression at the ECtHR, complementing classification-heavy resources (LexGLUE, LEXTREME, LEXam) and rule-bounded numerical tasks like CAIL sentencing; Section 1-2 argue the gap convincingly.
Rigorous, empirically motivated leakage control
Award-related material (Art. 41 claims, just-satisfaction reasoning, operative provisions) is withheld from inputs (Sec. 3.2, Fig. 4); targets pass eight validation checks with a manually audited bundled-proxy exception (72 of 230 candidates admitted, App. B.3). The Expanded-ReAct diagnostic (Table 5: MAE 18,220 to 8,810; Zero-F1 0.002 to 0.997) empirically demonstrates that leaked material would halve error, validating the design.
Honest statistical treatment of model comparisons
Diagnostic depth grounded in legal doctrine
Transparent limitations
Weaknesses
Thin headroom over trivial baselines
Zero/positive subtask is close to unlearnable by design
Label noise from LLM-assisted extraction is unquantified
Nominal euros plus large temporal shift complicate interpretation
Single-run LLM/agent evaluations
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ECtHR-NPD benchmark?
A public dataset of 14,575 European Court of Human Rights judgments labeled with the case-level non-pecuniary damage award (in euros) granted under Article 41. Models must predict the amount from case metadata, facts, violated Convention articles, and macroeconomic indicators, while the applicants' compensation claims and the Court's just-satisfaction reasoning are deliberately withheld so the answer cannot be copied from the text.
How well do AI models predict ECtHR compensation amounts?
Poorly. The best system (CatBoost on structured features) has a mean absolute error of about 9,881 EUR, only roughly 10% better than always predicting the training-set median (11,006 EUR). Prompted LLMs such as GPT-5.4 and Qwen3.5, and tool-using agents, performed worse, and every model family failed to recognize zero-award cases and to calibrate large awards; errors roughly doubled on the hardest (Challenging) subset.
Why is predicting ECtHR damages hard for large language models?
Awards are zero-inflated and heavy-tailed, decided on an equitable basis with no published formula, and the strongest predictors - the applicant's claim and the Court's Article 41 reasoning - are intentionally hidden. LLMs also process numbers as token sequences and handle magnitude poorly: in the paper's examples, models linearly scaled awards by applicant count, predicting 1.5 million EUR where 7,500 EUR was awarded. When the hidden material is given to agents, error nearly halves.
Is the ECtHR-NPD dataset and code publicly available?
Yes. Data is released on Hugging Face and code and documentation on GitHub. The release contains validated targets, split and diagnostic-view tags, structured features, and public HUDOC case identifiers so researchers can retrieve judgments from official sources; raw judgment text is not redistributed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review is based on the embedded extracted text only; Figures 1-5 were interpreted from captions/descriptions, not visual inspection.
- Appendix subsections C.4 (Retrieval Baselines) and C.5 (Encoder LMs) appear as headers without body text in the extraction, so some implementation details were unverifiable.
- Claims about specific model versions and snapshots (GPT-5.4, Qwen3.5, MiniMax-M2.7, March 2026 HUDOC snapshot) could not be externally verified and rely solely on the paper's statements.
- The paper's own reproducibility caveat applies: prompted LM and agent results are single runs, so reported numbers may vary on re-execution.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.