When Can a Machine Trust a Statute? A Survival Certificate for Machine-Extracted Legal Logic
cs.AI · cs.CL
Quick verdict
Paper proposes a Wilson-LB survival certificate for machine-extracted statutory implications: preregistered, transparent, but fragile and India transfer fails.
Summary
The paper asks how much formal logic extracted from statutes survives the extraction process itself. Two independently written rule-based parsers produce a 12-attribute binary context over Missouri statutes (29,365 sections, 458 chapters) and 15 Indian central Acts (502 sections); per-attribute disagreement (numthresh false-negative rate 0.429) calibrates a Monte Carlo error model (N=1000) replayed against the Duquenne-Guigues implication basis. An implication is certified when a one-sided Wilson 95% lower bound on survival reaches 0.95, with a Benjamini-Hochberg companion; each certified implication ships with premise spans and minimal counterexamples. A preregistered held-out gate passes (10 of 146 chapters across 7 Titles under exact semantics; 16 across 11 Titles with 5% tolerance), yet 93.2% of held-out chapters fall below an informativeness floor under the globally transferred error model, and a 2x2 factorial assigns this to calibration-rate transfer, not selection. Per-chapter calibration lifts the held-out mean certified fraction from 0.038 to 0.128. Replication on IndiaCode fails (0 of 6 eligible Acts in band) and is reported plainly. Main limitations: the reference parser is documentary rather than validated ground truth; disagreement plausibly understates real error (shared schema and vocabulary); per-attribute independence is assumed; and preregistration is only locally hash-frozen without third-party timestamping. Appendix K documents a retracted estimator and a substituted gate from an earlier revision.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Support does not predict robustness of machine-extracted legal implications; the interaction between an implication's attribute footprint and measured extractor error does (§5.1: support-792 implication at survival 0.000 vs support-12 at 0.965).
- A Wilson lower-bound survival certificate at N=1000 identifies a small auditable core of Duquenne-Guigues implications that passes a preregistered held-out gate (10/146 chapters across 7 Titles exact; 16/146 across 11 Titles tolerant).
- Global deployment fails by mechanism, not by overfitting: 93.2% of held-out chapters fall below the informativeness floor, and the preregistered 2x2 factorial assigns this to calibration-rate transfer across heterogeneous chapters, not selection.
- Per-chapter calibration materially improves informativeness (held-out mean 0.038 -> 0.128; in-band share 6.9% -> 52.7%), making per-chapter or tolerant deployment the only evidence-supported mode.
- The certificate is jurisdiction-conditional: it fails to transfer to IndiaCode (0 of 6 eligible Acts in band), and every certified implication remains a statistical bound under measured disagreement, not a guarantee of legal correctness.
Strengths
Preregistered held-out design with factorial deconfounding
§4/§5.3 and Table 3: the 2x2 cleanly separates calibration-rate transfer (+0.090 held-out, +0.070 design mean lift) from family selection (-0.031/-0.051, negative direction), falsifying the post-selection explanation for the 93.2% below-floor figure with a preregistered protocol and disjoint calibration half.
Radical transparency about failures and about its own history
IndiaCode 0/6 in band (§5.5), 93.2% below floor with CI [0.879, 0.962] (§5.2), and Appendix K documents a retracted invalid estimator (Method D), an earlier gate substitution, and a red-team reproduction with zero SHA-256 mismatches - negative results are archived, not scrubbed.
Negative controls establish the mechanism
The support-vs-survival separation is concrete and explained
Disclosed decision-rule sensitivity
Weaknesses
Certificate bounds an unvalidated quantity
Thin favorable core in the deployable default
Independence assumptions only partially stress-tested
Cross-jurisdiction validity collapses to Missouri
Preregistration is self-attested
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses a narrow, well-posed question: when the Duquenne-Guigues (DG) canonical implication basis is computed from a machine-extracted binary context of statutory attributes, which implications remain valid under measured extraction disagreement? Scope discipline is a model for this genre: the authors repeatedly separate Pr[implication survives measured disagreement] from Pr[extraction is legally correct] (§1 Scope, §6.2), and the certificate is explicitly conditional on the disagreement model. Central claims: (1) support does not predict robustness (§5.1); (2) a Wilson-lower-bound certificate identifies a small auditable core passing a preregistered held-out gate (10/146 exact, 16/146 tolerant across 11 Titles); (3) global deployment fails - 93.2% of held-out chapters below the informativeness floor - and a 2x2 factorial attributes this to calibration-rate transfer, not selection (§5.3); (4) per-chapter calibration lifts the held-out mean from 0.038 to 0.128, but transfer to IndiaCode fails outright (0/6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a survival certificate in this paper?
A per-rule statistical statement: a formal implication (rule like 'deadline implies numeric threshold') extracted from statute text is certified when its probability of surviving simulated parser disagreement has a one-sided Wilson 95% lower bound of at least 0.95, estimated from 1,000 Monte Carlo replays of measured error rates between two independently written extractors. Each certificate also names the premise spans and the cheapest perturbation that would kill the rule.
Did the method work?
Partially. On Missouri statutes it passed its preregistered held-out gate (10 of 146 chapters in the informative band under exact semantics; 16 of 146 with 5% tolerance), but 93.2% of chapters fell below the informativeness floor under one globally shared error model, and it failed entirely on Indian legislation (0 of 6 eligible Acts in band). The authors' conclusion: deploy per-chapter-calibrated or error-tolerant, never with a single global error model.
Why do two legal parsers disagree so much?
On Missouri's statutes, the independently written parser missed numeric thresholds - including spelled-out numbers like 'twenty-six weeks' - on 42.9% of the sections where the reference parser found one. Other attributes were far more stable: prohibitions (prohib) and resolved cross-references showed zero measured false negatives, which is why rules built on them survive easily.
Is the code and data available?
Yes. The abstract and conclusion state that code, data products, and the audit trail are released. The audit appendix (K) even retains a disowned, invalid estimator from an earlier revision as a labelled ablation, alongside a red-team reproduction reporting byte-identical headline outputs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (1-9) are available only as captions and embedded axis labels in the extracted text; plotted values cited in this review are taken from tables and body text, not visual inspection of the plots.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.