Structured Decomposition for Reliable LLM-Generated Access Control Policies
Vatsal Gupta, Darshan Sreenivasamurthy
cs.AI · cs.CR
Quick verdict
Structured pipeline lifts correct Rego generation to 50.3% vs 15.3% (372 ACRE); strong coverage gain, but security claims rest on n≤9 and self-derived tests.
Summary
The paper presents an industry (Apple) LLM pipeline that translates natural-language access control statements into executable Rego policies for Open Policy Agent. The pipeline detects policies, resolves coreference, segments multi-policy inputs, extracts Decision/Subject/Action/Resource/Condition/Purpose components, validates them against organization schemas injected into the generation prompt, and synthesizes deny-by-default Rego refined via up to three compile/lint feedback iterations. Three deterministic, non-LLM mechanisms — redundant-block consolidation, ownership/status semantic guardrails, and condition-aware test repair — supplement generation. On 372 ACRE-complete statements (filtered from 485), evaluated against a direct single-prompt Claude baseline (temperature 0.0), the system reaches 50.3% 'effective correct coverage' — compile plus Regal lint plus positive and negative test pass — versus 15.3% for the baseline (3.3×); on the 77-policy overlap, full correctness rises from 74.0% to 90.9%. Security metrics improve sharply: deny-rule completeness 87.5% vs 37.5% (n=8), ownership conditions 100% vs 40% (n=5), status conditions 100% vs 55.6% (n=9). Main limitations: correctness is measured by tests derived from the system's own extractions rather than independent semantic ground truth; security subgroup samples are tiny with no confidence intervals; 166 accepted-by-ACRE statements are rejected without error analysis; no ablations or prior-system comparisons; code is only promised upon acceptance, with the model version and prompts unspecified.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A modular pipeline of decomposition, schema-aware validation, and multi-stage verification achieves 50.3% end-to-end correct Rego on 372 ACRE-complete policies versus 15.3% for a direct single-prompt Claude baseline, a 3.3× improvement (Table 1).
- On the 77 policies both arms attempt, full four-criterion correctness rises from 74.0% (baseline) to 90.9% (system) (Table 2).
- Security semantics improve markedly: deny-rule completeness 87.5% vs 37.5% (n=8), ownership conditions 100% vs 40% (n=5), status conditions 100% vs 55.6% (n=9) on the overlap set (Table 3).
- Deterministic post-generation guardrails eliminate ownership- and status-condition omission (to 0%) on the evaluated dataset (§4.4).
- The primary challenge in LLM policy generation is reliable interpretation and validation of inputs, not Rego code synthesis (§1).
Strengths
Strict deployability criterion
Correctness requires all four of opa compilation, Regal lint, positive-test pass, and negative-test pass (§3.3, Table 1) — an operational standard stricter than the string-match or element-extraction metrics common in NLACP work, and the paper explicitly disclaims full semantic equivalence.
Head-to-head overlap analysis separates quality from coverage
Table 2 restricts to the 77 policies both arms attempt (74.0% → 90.9% all-four pass) and explicitly attributes the baseline's 100% compile rate to selection bias rather than claiming uniform superiority — an unusually honest framing.
Concrete security failure-mode taxonomy with quantified prevalence
Deterministic mechanisms outside the LLM path
Transparent reporting of unfavorable numbers
Weaknesses
The correctness criterion is self-referential
Security headline numbers rest on n=5–9 subsets
No ablations and no prior-system baselines
44.6% rejection rate unaudited
Reproducibility currently blocked
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This industry paper (Apple; arXiv 2609.24036) presents a modular LLM pipeline converting natural-language access control policies (NLACPs) into executable Rego for Open Policy Agent. The pipeline (§4) chains policy detection with coreference resolution and segmentation, DSARCP component extraction (Decision/Subject/Action/Resource/Condition/Purpose), schema validation against organization vocabularies injected verbatim into the generation prompt, and deny-by-default Rego synthesis with up to three compile/lint (opa check + Regal) repair iterations. Three deterministic, out-of-inference-path mechanisms are claimed as contributions: redundant-block consolidation, ownership/status semantic guardrails, and condition-aware test repair. The central claim (§1.1, §6) is a 3.3× improvement in 'effective correct coverage' over a direct single-prompt Claude baseline on 372 ACRE-complete statements (50.3% vs 15.3%, Table 1), plus stronger security semantics (deny 87.5% vs 37.5% with n=8; ownership 100% vs 40% with n=5; status 100% vs 55.6% with n=9, Table 3). A welcome secondary claim: the binding constraint is input interpretation and validation, not code synthesis (§1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper's system actually do?
It converts natural-language access control statements (e.g., 'nurses can read prescriptions') into executable Rego policy code for Open Policy Agent using a multi-step LLM pipeline: component extraction, schema validation, code generation with lint/compile feedback loops, and auto-generated unit tests — instead of one direct prompt.
How much better is it than just asking an LLM directly?
On 372 public ACRE access-control statements, 50.3% of generated policies passed all four correctness checks (compile, lint, positive and negative tests) versus 15.3% for a single-prompt Claude baseline — a 3.3× gap, driven mostly by the baseline failing to produce usable output. On the 77 statements both approaches attempted, full correctness was 90.9% vs 74.0%.
Is the code or dataset available?
The evaluation uses the public ACRE dataset (Slankas & Williams, 2013), but the tool's code is only promised 'upon acceptance' of the paper; as of this preprint, neither code, prompt templates, nor the exact model version are published.
What are the main caveats about the results?
The security-critical wins are measured on very small subsets (5–9 policies each) with no statistical uncertainty reported, and the automated tests defining 'correct' are derived from the system's own intermediate extractions — so they verify internal consistency more than true agreement with the original policy's meaning.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 (architecture diagram, per-module inputs/outputs example) are referenced but carry no analyzable content in the extracted text.
- Appendix A.1's worked NLACP→Rego example is described as a diagram; its content is missing from the extraction.
- The exact Claude model version, prompt templates, and dataset schemas are not present in the text, so some setup details cannot be verified.
- Review covers arXiv v1 (21 Sep 2026 preprint); the paper is not yet peer-reviewed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.