Auditability Is Not One Property: Rule Overlap, Behavioural Agreement, and Composition in Reinforcement Learning
cs.LG · cs.AI
Quick verdict
Decomposes RL auditability into six testable predicates; unusually honest negative results, but positive fusion gains are comparator-dependent on toy tasks.
Summary
The paper asks whether independently trained RL policies can be described and composed through auditable discrete behavioral rules, and decomposes auditability into six separately testable predicates: trace integrity, lossless coding, rule coverage, behavioural agreement, composition quality, and value-model reliability. Using a frozen shared quantile symbolizer, passive rule extraction anchored to a hash-bound ledger with exact replay, and offline confidence-ranked arbitration with a blind-spot fallback, it evaluates eight-seed REINFORCE populations on CartPole-v1 and Acrobot-v1. Key findings: rule-set overlap (Jaccard 0.56) does not predict behavioural agreement (51.4%, at the action-frequency floor); the CartPole fusion gain over a post-hoc comparator (+42.31, CI [23.98, 61.93]) is exploratory, reversing to -294.04 against the best held-out actor while a plain logit ensemble saturates the ceiling; the Acrobot failure is traced to an induction/deployment protocol mismatch, with greedy re-induction reversing the ordering (-141.77 vs random -234.26); and a fitted-Q GPI baseline collapses in both environments. A failure-driven design history documents harm from rule-constrained training, threshold collapse, and ledger-schema defects. Limitations: two toy benchmarks, eight seeds, unregistered predictions, a possibly misconfigured FQE baseline, and untested cross-hardware replay. Code and a 4.7 GB archived dataset are released.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Auditability of an RL policy decomposes into six separately testable predicates, and passing one (e.g., rule coverage) provides no evidence for another (e.g., behavioural agreement).
- Cross-seed rule-set overlap (mean pairwise Jaccard 0.557) does not imply behavioural agreement, which sits at 51.38% on fresh CartPole states — indistinguishable from the 51.32% action-frequency floor.
- The apparent Acrobot fusion failure is an induction/deployment protocol artifact: re-inducing rule banks from argmax-consistent labels reverses the arbitration ordering (-141.77 for confidence-ranked vs -234.26 for random).
- A fitted-Q GPI baseline collapses in both environments, which the authors use to bound — not support — any claim that rule fusion is superior to value-based composition.
- The one positive composition result (+42.31 on CartPole-v1, CI [23.98, 61.93]) is exploratory because the comparator was selected post hoc and the task exhibits ceiling effects.
Strengths
Predicate decomposition with strict evidentiary layering
Table 1 defines six predicates with explicit 'what it does not establish' boundaries; Section 2.3 states non-claims up front; Table 13 separates operational-health, diagnostic, and identification-bearing results — the framing prevents the common over-read of passing a mechanism check as passing a predicate.
Rare methodological candor, operationalized
Section 5 documents each failure (lossy grammar, rule-constraint damage, 100% blind-spot collapse at τc=0.90, ledger schema defect) with cause and fix; Table 4 registers predictions and marks P5, P7, P9 as contradicted and P6 as comparator-dependent; Section 7.7 steelmans the strongest objection; Section 6.3 includes an actual tamper probe that validates hash-chain detection rather than assuming it.
Controlled, paired experimental design
Root-cause isolation of the Acrobot failure
Strong reproducibility infrastructure
Weaknesses
Headline positive result is comparator-dependent and ceiling-saturated
FQE-GPI collapse may reflect a broken comparator, not a property of value-based composition
The central null is confounded by the coarse symbolizer
Degenerate Acrobot regime weakens several headline deltas
Limited external validity and self-referential framing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper reframes auditability of RL policies as six separately testable predicates — trace integrity, lossless coding, rule coverage, behavioural agreement, composition quality, and value-model reliability (Table 1) — and asks whether independently trained policies can be described and composed through discrete condition-action rules. Three claims structure the work: (i) the central negative result that rule-set overlap does not imply behavioural agreement; (ii) that Acrobot's apparent fusion failure is an induction/deployment protocol artifact, reversed by protocol-consistent re-induction; (iii) that a fitted-Q GPI baseline collapses in both environments, bounding any claim that rule fusion beats value-based composition. The single positive composition result (+42.31 on CartPole-v1) is explicitly flagged as exploratory. Scope is bounded to two Gymnasium tasks, eight seeds each, one frozen quantizer per task, and one hardware/software configuration; the non-claims section (2.3) is unusually thorough.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
That RL 'auditability' is not one property but six independently testable predicates. Its central empirical finding is that symbolic rule overlap between independently trained policies does not predict behavioral agreement (51.4% on fresh states, at chance level), and that an apparent composition failure on Acrobot was a protocol artifact of inducing rules from sampled actions while deploying under argmax.
Did rule fusion beat single policies?
Only in an exploratory sense: on CartPole fusion scored +42.31 over the single actor with the highest training return, but four of eight actors reached the 500 ceiling, a plain logit ensemble scored 500, and fusion was 294 points below the best held-out actor. The authors themselves frame this as a statement about comparator selection, not fusion strength.
Is the code and data available?
Yes. The paper reports a public GitHub repository (cyrilliu1974/AuditableRL) and a figshare archival dataset (584 files, 4.7 GB uncompressed, with a SHA-256 checksum file), plus a compact result package that maps each reported headline number to its source artifact.
What environments and algorithms were used?
Gymnasium CartPole-v1 and Acrobot-v1 with eight REINFORCE seeds each (2-hidden-layer, 64-unit networks). Policies were discretized by a frozen four-bin quantile symbolizer, converted to condition-action rule banks, and fused by confidence-ranked arbitration with a blind-spot fallback; evaluation used 100 matched held-out episodes per policy.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extracted text omits portions of Section 3 (related work), mid-Section 5.3, the Section 7.4 opening, and Tables 8 and 14; figures are available only as captions, so visual claims (Figures 3-7) could not be checked.
- The metadata abstract is truncated mid-sentence; claims were assessed against the full-text abstract instead.
- Numbers were verified only against the provided text; the released repository and 4.7 GB dataset were not accessible for verification.
- Several key analyses (A1b, A2) are referenced as appendix/analysis artifacts not fully included in the extraction, limiting assessment of their internal rigor.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.