Who Audits Whom, on What Substrate, with What Evidence? An Independence-Graded Audit Protocol for Agentic AI
cs.AI · cs.CR · cs.ET
Quick verdict
A three-axis independence grading scheme for AI audits, formally grounded and honestly caveated, but validated only by simulation with declared parameters.
Summary
The paper argues that independence in agentic-AI auditing should be graded, not declared, along three axes: principal (who controls the auditor), substrate (shared model family, toolchain or guardrails) and evidence (attestable vs self-reported). Each axis is scored 0-3; the audit grade is their minimum. The substrate axis is grounded by transplanting the beta-factor model of common-cause failure from reliability engineering, yielding propositions that same-substrate panels plateau at a common-cause floor (Prop 4.2) and that evidence grades form a strict chain (Prop 4.3). A seven-step protocol (Table 3, Algorithm 1) specifies third-party-verifiable outputs, caps substrate grade at 1 when lineage is undisclosed, and reserves the opinion for a named human signatory. A procurement-agent case study (Table 4) and a 200,000-trial Monte Carlo study (Table 5) quantify the model: a conventional internal audit using a same-family second agent on provider logs surfaces 5.9% of faults in two classes and none in the other two, while an external cross-substrate attested audit reaches 77.3% worst-case; a calibration against a measured judge panel implies beta ~0.46, an order of magnitude above IEC 61508's band. Mapping to EU AI Act, ISO/IEC 42006 and UK guidance is concrete. Main limitations, largely self-declared: no empirical evaluation on real agents; beta factors and reporting probabilities are declared, not estimated; the calibration transfers from evaluation panels to audits; and the minimum rule ties the mean on adversarial assurance.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Independence in agentic-AI auditing must be graded on three orthogonal axes - principal, substrate, evidence - aggregated by weakest link, with the same rubric applied when the auditor is itself an agent.
- Transplanting the beta-factor common-cause model gives the substrate axis a formal basis: escape probability plateaus at gamma_D no matter how many same-substrate auditors are added, and one cross-substrate auditor dominates the panel whenever pA*pB < gamma_D (Prop 4.2).
- Under the declared-parameter model, a conventional internal audit of an agent (real team, second same-family agent, provider logs) surfaces 5.9% of catchable faults and nothing in half the fault classes - a structural, not diligence, failure.
- The minimum is the right aggregate to publish because it rarely overstates assurance (26.5% of triples, median overstatement 0) versus mean or weighted sums (92.2%/90.8%, median 0.33), though it is not the sharpest predictor of adversarial assurance.
- Instruments in force (EU AI Act Arts. 12/14/26/43, ISO/IEC 42006, UK AI Risk Management Toolkit, NYC Local Law 144) regulate only principal independence and are silent on substrate and evidence, which the triple supplies as reportable metadata.
Strengths
Genuinely new joint contribution, cleanly positioned
Table 1 shows no prior line grades all three independence properties on one audit, aggregates by weakest link, and applies the rubric symmetrically when the auditor is itself an agent; the closest neighbour (ref [41]) is treated as a special case (Eq. 2 applied to judge panels) rather than displaced.
Formal machinery with verifiable internal consistency
The beta-factor transplant is not decorative: the calibration arithmetic checks out (neff=2 at n=9 implies rho~0.44, and inverting Eq. 4 at p=0.10 gives gamma_D~0.046, beta~0.46), and the closed forms reproduce every Table 5 cell (e.g., 9.9% catch at S=0 x 0.6 reporting at P=1 = 5.9%; 85.9% x 0.9 = 77.3% for C), so the quantitative story coheres end-to-end.
Unusual candour about what the results do and do not show
Operational, third-party-checkable protocol
The dominance result is decision-relevant and memorable
Weaknesses
Entirely analytical and simulated; no real audit is studied
Load-bearing parameters are stipulated, and only two of four beta values have any anchor
Fault-class selection pre-loads the headline asymmetry
The weakest-link justification is partly post hoc
Substrate rubric granularity is a judgment call the model cannot yet support
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper propose?
It proposes that AI audits report three graded scores (0-3) instead of a binary 'independent' label: principal independence (who controls and pays the auditor), substrate independence (whether auditor and auditee share a model family, toolchain, guardrails or hosting that fail together), and evidence independence (whether records survive tampering by the agent, operator or provider). The overall audit grade is the minimum of the three, and the same rubric applies when the auditor is itself an AI agent.
What is the 5.9% finding?
In a Monte Carlo simulation - not a field study - an internal audit that uses a real audit team, a second agent on the same model family, and provider-controlled logs surfaces and reports only 5.9% of the faults it could in principle see in two fault classes, and 0% in two classes that require tamper-evident records. The author declares all simulation parameters rather than measuring them, so the number illustrates a model's implication and should not be read as a measured detection rate for any real system.
Why can't adding more same-substrate auditors help?
Under the common-cause (beta-factor) model borrowed from reliability engineering, auditors sharing the auditee's model family, toolchain or guardrails tend to miss the same hard inputs. Escape probability plateaus at the common-cause rate no matter how many auditors are added, whereas one genuinely cross-substrate auditor reaches the fully independent bound (Proposition 4.2). The paper computes this floor as roughly 9% for identical substrates versus 1% for two independent reviewers on its parameters.
Is the paper's code and data available?
Yes. A small Python package (NumPy/SciPy only) with two experiment drivers and eighteen unit tests reproduces every reported number from a fixed seed in under two minutes. It is deposited on Zenodo (DOI 10.5281/zenodo.22768569). There is no external dataset: the study generates its own random draws.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 were assessed from captions and in-text descriptions only; the rendered graphics were not inspectable.
- The Zenodo artifact (DOI 10.5281/zenodo.22768569) was not executed or inspected; reproducibility judgement rests on the paper's description.
- Several cited 2026 policy documents (UK AI Risk Management Toolkit, Digital Omnibus adoption, FRC and PCAOB items) could not be verified against sources in this offline review.
- This is an arXiv preprint in ACM-letter format by a single author; no prior peer review is evident.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.