Mining Legal Arguments in U.S. Corporate Case Law
Luis Brena, William Jurayj, Gregory Deyesu, Zaid Al-Huneidi, Andrew Blair-Stanek, Benjamin Van Durme, Benjamin Van Durme
cs.CL
Quick verdict
First expert-annotated tree-structured argument corpus for U.S. §368 tax cases; labels reliable, support edges weak, cross-case retrieval generalizes poorly.
Summary
The paper introduces an expert-annotated corpus of 42 U.S. federal tax opinions on corporate reorganizations under I.R.C. §368 (1935–1987), claimed as the first tree-structured argument-mining resource for U.S. tax case law. Two law students annotated free-form spans with five functional labels (Rule, Analysis, Conclusion, Background Facts, Procedural History) and directed support links forming argument trees; a law professor adjudicated by selecting complete annotation files. The release includes span, sentence, flat, and tree views (718 explicit spans, 644 edges, 82 implicit intermediate conclusions). Ten double-annotated cases show functional labels are the most reliable layer (αu 0.45–0.88, macro 0.65), while implicit-node insertion (κ slightly negative) and directed edges (P+ 0.06–0.16 one-to-one) are weak; path reachability agrees better (κ 0.59 relaxed). Classification under case-disjoint five-fold evaluation shows labels are learnable: LegalBERT reaches 0.71 Macro-F1 (five classes), 0.80 (four), GPT-5.4 with context 0.79/0.84; Rule–Analysis confusion dominates. In masked-slot retrieval (490 queries), fine-tuned ModernBERT improves within-case ranking (80.45% Hit@20, same-case filtered) but BM25 leads fold-global (42.49% vs ≤15.67% for fine-tuned), showing weak cross-case generalization; structured queries show no reliable advantage (bootstrap CIs include zero). Main limitations: small, temporally dated corpus; low-reliability structural layer; adjudication selects one annotator's interpretation.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The paper releases the first expert-annotated, tree-structured argument-mining corpus for U.S. federal tax reorganization opinions under I.R.C. §368.
- Functional node labels are the most reliable annotation layer (macro αu 0.65), while directed support edges and implicit intermediate conclusions show weak agreement; directed-path reachability is more stable than exact local edges.
- Functional labels are learnable under case-disjoint evaluation: LegalBERT reaches 0.71 Macro-F1 (five classes) and GPT-5.4 with case context 0.79, with Rule–Analysis the main confusion.
- Supervised fine-tuning improves within-case masked-slot retrieval (80.45% Hit@20 in the same-case filtered pool) but cross-case generalization remains weak, with BM25 leading fold-global evaluation (42.49% Hit@20) and structured queries showing no reliable advantage.
Strengths
First-of-kind expert resource for an underrepresented U.S. domain
42 §368 opinions (1935–1987) annotated by two law students and adjudicated by a tax-law professor (§3); Appendix O systematically positions it against ECHR/CJEU/Indian/U.S. corpora, none offering expert-adjudicated document-level support trees for U.S. tax law.
Exemplary reliability transparency
Four agreement layers (§3.4): functional labels macro αu 0.65 (Table 1), directed-edge κ 0.02–0.12 under 1:1 pairing (Table 12), implicit-node insertion κ ≈ −0.02 to −0.04 (Table 2), full-transitive path κ 0.59 (Table 11) — and the paper downgrades its own tree layer to 'exploratory' rather than overselling it.
Leakage-controlled evaluation with real uncertainty analysis
Full experimental specification enabling reuse
Weaknesses
Small, temporally stale corpus
The novel layer is the least reliable
Cross-case retrieval collapses and is undiagnosed
No code release; classification uncertainty unreported
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper contributes an expert-annotated dataset of 42 U.S. federal tax opinions on corporate reorganizations under I.R.C. §368 (subtypes A–D and F; E and G excluded), spanning 1935–1987, with five functional span labels (Rule, Analysis, Conclusion, Background Facts, Procedural History) and directed support links organized as syllogistic argument trees, including implicit intermediate conclusions ('enthymemes'). The authors claim three contributions (§1): the first expert-annotated, tree-structured argument corpus for this domain; multiple representations (span, sentence, flat, tree) with reliability statistics; and case-disjoint classification and retrieval experiments showing labels are learnable while structure-aware retrieval generalizes weakly. The narrow scope is defensible: §368 doctrine forces courts to connect statutory tests, transaction-specific facts, and intermediate conclusions, making it a dense testbed for argument structure.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Mining Legal Arguments in U.S. Corporate Case Law dataset?
It is a corpus of 42 U.S. federal tax opinions on corporate reorganizations under I.R.C. §368 (published 1935–1987), expert-annotated with five functional labels (Rule, Analysis, Conclusion, Background Facts, Procedural History) and directed support links forming argument trees, released on Hugging Face in span, sentence, flat, and tree formats.
How reliable are the annotations in this corpus?
Functional labels are the most reliable layer: unitized Krippendorff's αu ranges 0.45–0.88 per label (macro 0.65) over ten double-annotated cases. Directed support edges (κ 0.02–0.12 under strict pairing) and implicit intermediate conclusions (κ slightly negative) are weak, so the authors recommend treating the tree layer as exploratory.
Can machine learning models learn these legal labels and retrieve missing arguments?
Yes for classification: under case-disjoint five-fold evaluation LegalBERT reaches 0.71 Macro-F1 (five classes) and GPT-5.4 with case context 0.79. For masked-slot retrieval, fine-tuning helps within a case (80.45% Hit@20) but generalizes weakly across cases, where BM25 remains best (42.49% Hit@20).
What are the main limitations of this dataset?
It is small (42 cases, 718 annotated spans), restricted to one narrow tax doctrine, and contains no opinions after 1987, so results may not transfer to modern case language, other tax doctrines, or other legal domains; the tree-structure layer also has low inter-annotator agreement.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 1 and Table 2 content was interleaved with body text during extraction; values were reconstructed from surrounding text and table captions, so minor ordering artifacts are possible.
- Figures 1–2 (overview diagram, Hit@k curves) are referenced but not analyzable from the extracted text.
- Some appendix details (e.g., exact classification split seed, Appendix B workflow imagery) are absent or only partially present in the extraction.
- Review is based on arXiv v1 (2609.25441); the Hugging Face data files themselves were not inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.