Policy-as-Skill: Governed LLM Decision Support with Evidence, Deterministic Control, and Audit
Kabeh Mohsenzadegan, Vahid Tavakkoli, Kyandoghere Kyamakya
cs.AI · cs.SC
Quick verdict
PaS packages governance as versioned executable skills; results are honest but headline gains ride a degenerate subset, synthetic corpus, single-expert labels.
Summary
The paper introduces Policy-as-Skill (PaS), an architecture that packages organizational policy as a versioned, executable capability—a ten-element tuple binding skill identity, version, retrieval scope, evidence requirements, decision schema, review triggers, audit fields, failure behavior, prompt template, and context—rather than relying on prompts or retrieval alone. Four variants (PaS Retrieval, +Controller, +Audit, Full) are evaluated against nine baselines, including LLM+RAG and Policy-as-Prompt, on 600 synthetic policy tasks (QA, compliance, risk classification, conflict detection) using a fixed Gemma4/Ollama backend. PaS+Audit achieves 53.8% exact accuracy and review F1 0.854 versus 49.8%/0.695 for LLM+RAG, with citation precision 1.000 and native audit completeness 1.000. The deterministic controller raises aggregate accuracy to 61.2%, but ablations show it degrades risk classification by 23.3 points and converts all conflict cases to needs-review; excluding that degenerate subset, PaS Full falls below LLM+RAG. Bootstrap CIs and exact tests (McNemar, paired) support the contrasts, and a 286-weighting sensitivity analysis shows the composite readiness ranking is unstable. Main limitations: an eleven-fixture synthetic corpus with single-expert labels, a benchmark that informed controller development (not held out), an empty manual annotation file, and one small CPU-run model. Honest provenance discussion and clean ablations are strengths; external validity on real policy corpora remains unestablished.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Packaging policy as a versioned executable skill (ten-element contract, Section III) yields strong human-review routing (review F1 0.854 vs 0.695 for LLM+RAG) and complete native audit records without overriding model decisions.
- A deterministic controller raises aggregate exact accuracy from 53.8% to 61.2% but its effect is strongly task-dependent — degrading risk classification by 23.3 points — so deterministic control should be selective, not universal.
- Primary metrics that credit no PaS-specific metadata show PaS+Audit beating LLM+RAG on governance and review metrics while exact-decision gains are modest (+4.0pp).
- Composite governance-readiness rankings are unstable to weighting choices (286 weightings flip the top method among PaS Full, Keyword Search, and PaS+Audit), so a single readiness score should not ground deployment claims.
- The evaluation is system-development-informed, not a frozen held-out test; a frozen, independently validated evaluation remains necessary before generalization or legal-correctness claims.
Strengths
Clean component ablation
Four PaS variants (Section III) separate skill-scoped retrieval, deterministic control, and audit validation; Section V-B decomposes the controller's 401/600 decision changes into 181 corrections, 137 regressions, and 83 lateral moves, and Table III isolates per-family effects including a statistically clear -23.3-point risk-classification drop (p = 1.27e-4).
Fairness-conscious metric design
Primary metrics (exact accuracy, macro-F1, review F1) credit no PaS-specific audit metadata; a common trace envelope scores 1.0 for all thirteen methods; and the weighted governance-readiness index is demoted to secondary, with a 286-weighting sensitivity analysis (Section V-C) showing the top method flips between PaS Full (35.1%), Keyword Search (33.9%), and PaS+Audit (30.8%).
Unusually honest provenance statement
Complete statistical reporting for key contrasts
Open artifacts
Weaknesses
Degenerate conflict subset inflates the headline
Tiny synthetic corpus with single-expert labels
Macro-F1 contradicts the accuracy narrative
Single small CPU backend
Evaluation is development-informed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper proposes Policy-as-Skill (PaS), an application-layer architecture in which an organizational policy becomes a versioned, executable capability: a ten-element tuple (Section III, Eq. 1, Table I) binding skill identity, version, retrieval scope, evidence contract, decision schema, review triggers, audit fields, failure behavior, prompt template, and context contract. Four research variants enable clean ablation — PaS Retrieval, PaS+Controller, PaS+Audit, PaS Full — compared against nine baselines (Direct/generic LLM, Keyword Search, Standard/Hybrid RAG, Hybrid RAG+reranker, LLM+RAG, PaP, Structured PaP) over 7,800 task-method runs. The central claims: (1) skill-scoped retrieval plus audit validation materially improves human-review routing (review F1 0.854 vs 0.695 for LLM+RAG) without crediting PaS-specific metadata; (2) a deterministic controller raises aggregate exact accuracy to 61.2% but is strongly task-dependent, motivating selective rather than universal rule-based intervention; (3) composite governance-readiness rankings are unstable to weighting choices, so a single score should not ground deployment decisions. The provenance claim is deliberately scoped: model-level unseen data, but system-development-informed evaluation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Policy-as-Skill (PaS)?
PaS is an architecture that treats an organizational policy as a versioned, executable 'skill': a ten-element contract binding the skill's identity, version, retrieval scope, required evidence, decision schema, human-review triggers, audit fields, failure behavior, prompt template, and context. This makes version control, evidence validation, review routing, and audit records explicit runtime artifacts instead of relying on a prompt or retrieval alone.
How much better is PaS than standard RAG?
On the paper's 600-task synthetic benchmark with a fixed Gemma4 backend, PaS+Audit reaches 53.8% exact accuracy and review F1 0.854, versus 49.8% and 0.695 for LLM+RAG. Adding the deterministic controller raises aggregate accuracy to 61.2%, but the paper's own analysis shows that gain comes almost entirely from a conflict-detection subset whose reference answer is always 'needs review'; excluding it, the controller variant scores 48.2%, below LLM+RAG's 50.4%.
Can the paper's results be reproduced?
The authors release code, benchmark, configuration, traces, and evaluation utilities at github.com/vtavakkoli/policy-as-skill, with a fixed backend (gemma4:e2b via Ollama), seed 7, and 1,000 bootstrap iterations documented; the run is CPU-scale. Main caveats: a single small model, and a benchmark whose traces informed controller development, so results are development-informed rather than held-out.
What are the paper's main limitations?
The policy corpus is eleven short synthetic fixtures labeled by a single expert with no adjudication; the benchmark was used during system development so it is not a held-out test; the manual evidence-faithfulness annotation file is empty; and all 7,800 runs use one small Gemma4 model on CPU, leaving generalization to stronger models and real regulations untested.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are referenced but only captions were extracted; visual content (accuracy-by-family charts, runtime diagram) was not assessable.
- Implementation details beyond the narrative (retrieval parameters, controller rules, corpus texts) may live in the released repository rather than the paper text.
- References dated 2025-2026 limit literature cross-checking from prior knowledge.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.