IntLawNER: A Named Entity Recognition Dataset and Benchmark in International Law
Genis Skura, Roland Bouffanais, Didier Wernli
cs.AI · cs.IR
Quick verdict
First international-law NER benchmark (2,987 sentences, 7 types); sound pipeline and error analysis, but Claude-silver circularity and rare-label sparsity.
Summary
The paper introduces IntLawNER, the first token-level NER dataset and benchmark for codified international law: 2,987 gold-annotated sentences and 8,094 entity spans from ICJ decisions, UN Security Council resolutions, and ECtHR judgments, with a seven-type schema (treaty, provision, resolution, domestic law, case citation, soft law, court instrument). A three-phase hybrid pipeline reduces 468k source sentences to 3,000 candidates: weak labeling plus Legal-BERT clustering for diversity, a LangGraph agentic loop (four nodes, ~800 Claude Opus 4.6 calls) that selects and silver-annotates, then human review, with 89.6% of gold spans accepted unchanged from silver. The silver-to-gold analysis shows Cohen's kappa of 0.964 on boundary-matched spans but macro-F1 of 0.753 under exact matching, with court-instrument F1 collapsing to 0.044 — evidence that aggregate agreement metrics can mask label-level failure. Benchmarking regex, GLiNER, nine LLMs, and fine-tuned transformers shows GLiNER collapsing (0.243 micro-F1), XLM-RoBERTa-large at 0.849, and Claude Opus 4.6 5-shot best at 0.873; contrastive few-shot examples improve every LLM (Sonnet 4.6 +14.8 points). Main limitations: gold derives from LLM silver labels via a single human review pass with no human-human IAA reported, the benchmark's top model belongs to the family that produced the silver labels, rare-label test sets hold only 18–37 spans, and the schema awaits expert legal validation.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- IntLawNER is the first token-level NER dataset and benchmark for codified international law, with 2,987 gold sentences and 8,094 spans across ICJ, UNSC, and ECtHR corpora and a seven-type entity schema.
- A hybrid algorithmic–agentic pipeline reduces 468,287 source sentences to 3,000 annotation candidates at ~$140–240, and 89.6% of gold spans were accepted unchanged from the LLM silver layer.
- Human–machine agreement metrics mislead in domain-specific NER: κ=0.964 on boundary-matched spans coexists with exact-match macro-F1 0.753 and IL_COURT_INSTR F1 of 0.044.
- Contrastive few-shot examples improve all nine evaluated LLMs over zero-shot, with Claude Opus 4.6 5-shot best at 0.873 micro-F1, ahead of fine-tuned XLM-RoBERTa-large at 0.849.
Strengths
Fills a genuine resource gap with an unusually complete specification
Prior legal NER benchmarks (Leitner 2019/2020, Kalamkar 2022, Angelidis 2018) cover only domestic law (§2); IntLawNER adds three international corpora and a seven-type schema anchored in ICJ Statute Art. 38(1), with a full label glossary including decomposition rules and exclusions (Appendix F) that most NER papers never release.
Exemplary silver-to-gold error accounting
§4.3/§5.3 decompose all human deviations (201 label-only, 289 boundary-only, 29 both, 325 added spans; Table 6) and show κ=0.964 on boundary-matched spans coexists with exact-match macro-F1 0.753 and IL_COURT_INSTR F1 0.044 — a concrete, quantified warning about aggregate agreement metrics for LLM-annotated datasets.
Self-audit of selection bias with data, not assertion
Above-average evaluation hygiene
Weaknesses
Gold layer is LLM-dominated with an unquantified human pass
Annotation-benchmark circularity is unaddressed
Headline per-label claims rest on tiny test samples
No contamination controls for the LLM benchmark
Confusing dual reporting basis for fine-tuned baselines
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper contributes both a resource and a benchmark: IntLawNER, presented as the first token-level NER dataset for codified international law — 2,987 gold-annotated sentences and 8,094 entity spans from ICJ decisions (CD-ICJ), UN Security Council resolutions (1946–2026), and ECtHR judgments, under a seven-type schema (Table 2; full glossary in Appendix F). Four claims are staked in §1: (i) the resource fills a real gap, since existing legal NER benchmarks (Leitner, Kalamkar, Angelidis, Aejas, Karamitsos) cover only domestic law; (ii) the hybrid algorithmic–agentic pipeline is cost-effective ($140–240 for ~800 LLM calls, Limitations); (iii) aggregate agreement misleads — Cohen's κ=0.964 on boundary-matched silver-vs-gold spans coexists with exact-match macro-F1 of 0.753 and 0.044 for IL_COURT_INSTR (Table 6); (iv) a system hierarchy in which GLiNER collapses (0.243 micro-F1), fine-tuned XLM-RoBERTa-large reaches 0.849, and Claude Opus 4.6 with 5-shot prompting is best at 0.873 (Table 3). The claims are mostly well-scoped and internally supported; the main interpretation risk is reading LLM superiority as legal reasoning when a label-provenance confound exists (see Risks).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is IntLawNER and what does it contain?
IntLawNER is described as the first named entity recognition (NER) dataset and benchmark for codified international law. It contains 2,987 human-reviewed sentences and 8,094 entity spans extracted from ICJ decisions, UN Security Council resolutions, and ECtHR judgments, labeled with seven entity types such as treaty, provision, resolution, domestic law, case citation, soft law, and court instrument. The annotation layer is released under CC-BY-4.0 on Hugging Face, with pipeline and evaluation code on GitHub.
Which model performs best on the IntLawNER benchmark?
According to the paper, Claude Opus 4.6 with 5-shot prompting achieves the best score at 0.873 micro-F1, narrowly ahead of Claude Sonnet 4.6 (0.871) and GPT-5.5 (0.861). The best fine-tuned transformer, XLM-RoBERTa-large, reaches 0.849, while the zero-shot span model GLiNER collapses to 0.243 and a regex baseline scores 0.345.
Why do the authors say Cohen's kappa is misleading for LLM-annotated NER datasets?
The kappa of 0.964 was computed only on spans whose boundaries already matched between the LLM's silver labels and the human gold. When missed entities, boundary errors, and label corrections are included under strict exact matching, macro-F1 falls to 0.753, and the court-instrument class drops to F1 0.044 — showing that high aggregate agreement can hide catastrophic per-label failures. The authors recommend per-label confusion matrices as a minimum reporting standard.
Where can I download IntLawNER and its code?
The dataset with train/dev/test splits, few-shot prompt templates, and annotation guidelines is at huggingface.co/datasets/Geo14/IntLawNER (annotation layer CC-BY-4.0; underlying legal texts keep their original ICJ/UN/CoE terms). The construction pipeline, evaluation scripts, and bootstrap intervals are at github.com/mrgeooo14/IntLawNER.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 (pipeline diagram, label distributions) are referenced but not analyzable from the extracted text.
- PDF extraction introduces minor typographic noise (ligatures, subscripts); numbers were cross-checked across tables where duplicated.
- The nine LLM evaluation rows cannot be independently verified from the text; no decoding parameters are reported.
- Appendix tables extracted densely; individual cell values were spot-checked but not exhaustively re-parsed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.