Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code
Jisoo Kim, Taeyoon Kwack, Jinwoo Jang, Woo Kyung Kim, Honguk Woo
cs.SE · cs.AI
Quick verdict
Regulation-to-code compliance checking with self-verified checklists; strong F1 gains (+12.3 PIPA), but LLM-judge circularity and GPT-5 data temper claims.
Summary
The paper introduces Code-as-Auditor, an LLM framework that reformulates regulatory compliance assessment as executable code generation. Via a generator-verifier self-verification loop, regulations are translated into a code template encoding an IRAC-style reasoning path — an applicability condition, a compliance condition, and effects handling cross-provision exceptions — organized as a hierarchical compliance decision tree. A case-agnostic checklist of atomic questions accumulates during generation; at inference, each item is rewritten into case-specific questions structured by Kelsen's four spheres of normative validity (personal, material, temporal, spatial) and answered with ternary True/False/None logic for record-bound adjudication. On the EU AI Act, GDPR, and a new 406-case PIPA dataset built from Korean PIPC deliberation decisions, with Qwen2.5-7B as inference backbone, the method reports the best F1 (90.7, 61.2, 55.3), exceeding the strongest baselines by 3.1 to 12.3 points; it degrades less at 0.5B-1.5B scale and edges the best reasoning baselines (GPT-5/GPT-5 Mini) by 1.3-1.4 points on LegalBench Telemarketing (92.0) and SARA (82.7). Ablations attribute the largest gains on complex regulations to checklist refinement and the code template, while removing self-verification yields non-executable code. Main limitations: the verifier belongs to the same GPT-5 family and certifies only well-formedness, not semantic fidelity to statutes; case summaries are GPT-5 compressions of adjudicated decisions that may smooth distractors; no significance tests are reported; and no code or data release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Translating regulations into executable decision-tree code via a verifier-guided loop yields higher article-level violation F1 than natural-language or other neuro-symbolic representations: +3.1 (EU AI Act), +3.4 (GDPR), +12.3 (PIPA) over the best baselines with Qwen2.5-7B inference (Table 4).
- Grounding checklist items through Kelsen's four spheres of normative validity outperforms Claim-Element, 5W1H, FActScore, and ClaimDecomp decompositions on real adjudicated PIPA cases (Fig. 7).
- The method is robust to inference-model scale (86.0 F1 at 0.5B on EU AI Act vs 27.2 for logic baselines, Table 5) and transfers beyond privacy law, edging GPT-5/GPT-5 Mini by 1.3-1.4 points on LegalBench Telemarketing (92.0) and SARA (82.7) (Table 6).
- Self-verification is a structural prerequisite — removing it produces syntactically invalid code and zero scores (Fig. 8) — while checklist refinement and the code template contribute most on complex regulations (recall -37.91 on PIPA; F1 -30.86).
Strengths
Principled, ablation-supported grounding ontology
Case grounding via Kelsen's four spheres of validity (§3.2) is evaluated head-to-head against Claim-Element, 5W1H, FActScore, and ClaimDecomp on real PIPA cases (Fig. 7): Kelsen wins all reported metrics, while Claim-Element reaches only 37.44 precision and FActScore 50.21 overall — rare rigor for a design choice usually asserted, not tested.
Complexity-scaled evaluation design
The three benchmarks are explicitly ordered by structural complexity (Tables 2-3: cross-references/article 0.61 → 0.80 → 1.12; conditional/deontic ratio up to 2.70), and the largest gain (+12.3 F1 over Semantic FOL 43.0 on PIPA, Table 4) appears exactly where structure is hardest, consistent with the claimed mechanism.
Strong small-model robustness
New real-world dataset from authoritative labels
Honest Discussion section
Weaknesses
Verifier circularity undermines the auditability claim
Statistical reporting is thin for the claimed margins
Data-construction circularity on the flagship benchmark
Underspecified execution semantics and reproducibility
Post-hoc explanation for over-iteration degradation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Code-as-Auditor (CIKM '26) positions itself against two failure modes: free-form LLM legal reasoning (fabricated knowledge, unsound chains — motivated by Fig. 1) and brittle neuro-symbolic rule induction. Its four claimed contributions are a code-based formalization of compliance reasoning (template tau = (C_app, C_com, epsilon) embedding IRAC, §3.1), verifier-guided regulation-to-code generation (Algorithm 1), Kelsen-grounded case refinement (§3.2), and a new 406-case PIPA dataset (§4). Headline results: best article-level F1 on EU AI Act (90.7), GDPR (61.2), and PIPA (55.3) with margins of +3.1/+3.4/+12.3 over the second-best method (Table 4, Qwen2.5-7B inference); robustness at 0.5B scale (Table 5); and LegalBench transfer wins of +1.3 (Telemarketing 92.0 vs GPT-5) and +1.4 (SARA 82.7 vs GPT-5 Mini) (Table 6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Code-as-Auditor?
It is an LLM framework that converts legal regulations into executable code — checklists plus a hierarchy of decision trees — so that whether a specific case complies is decided by running that code against case evidence, rather than by free-form language-model reasoning. Each provision is encoded with an applicability condition, a compliance condition, and effects for exceptions, and checklist questions are grounded via Kelsen's four dimensions of legal validity (who, what action, when, where).
How much better is it than existing methods?
With Qwen2.5-7B as the inference model it reports the best article-level F1 on three benchmarks — 90.7 on the EU AI Act, 61.2 on GDPR, and 55.3 on its new Korean PIPA dataset — beating the strongest baselines by 3.1 to 12.3 points. On two LegalBench transfer tasks it reaches 92.0 (Telemarketing) and 82.7 (SARA), about 1.3-1.4 points above the best frontier-model baselines (GPT-5 / GPT-5 Mini), though those margins fall within reported variance.
What data was it evaluated on?
Privacy benchmarks from PrivaCI-Bench covering the EU AI Act and GDPR, plus a new dataset of 406 real deliberation decisions from Korea's Personal Information Protection Commission (PIPC) with compliance labels taken from the official adjudications. Transfer tests used LegalBench's FTC Telemarketing Sales Rule and SARA US federal tax law entailment tasks.
What are the main limitations?
The code verifier is itself an LLM from the same GPT-5 family, so it checks well-formedness rather than semantic fidelity to the statute; case inputs are GPT-5-compressed summaries of already-adjudicated decisions, which the authors admit may smooth out hard distractors; results are reported without significance testing; and no code or data release is stated in the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 5-9 are not analyzable from the extracted text; several cited values (ablation deltas, Fig. 7 comparison numbers) come from body text rather than figure data.
- Prompt tables are explicitly 'abbreviated' (Table 1), so exact generation and verification prompts are unavailable for assessment.
- The paper text appears complete through the references with no visible appendix; if supplementary material exists, it was not part of the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.