From Capability to Assurance in Autonomous Penetration-Testing Harnesses: A Framework and Reference Implementation
Joas Antonio dos Santos Barbosa
cs.CR · cs.AI
Quick verdict
Sound code-enforceable assurance framework for LLM pentest agents with honest negative results, but flagship run predates its own instrumentation.
Summary
The paper argues that autonomous LLM penetration-testing agents are judged almost exclusively by capability (flag capture, PoC reproduction) while assurance—whether findings are true, authorized, and auditable—is neglected, and claims assurance is a harness property enforceable in code. Section IV defines five properties, each with a formal model and acceptance test: evidence grounding (P1), non-destructive claim reduction (P2), evidence-graded CVSS v3.1 severity with a cap table (P3), attributable authorization via signed ns-cap.v1 tokens (P4), and externally anchored hash-chained audit (P5). Table I codes five systems, mostly with absence-of-claim cells, by a single coder. The reference implementation is the author's own NeuroSploit Rust harness (pinned commit; Table IV: 23,370 LOC, 312 tests, 27 validators over 31 CWEs). The deterministic P4/P5 battery (Table III) correctly rejects tampered/wrong-key/expired tokens but finds an off-grant primary target was reconned and logged allow—a genuine enforcement gap, so P4 is partial; P5 anchoring is design-only. The archived run (Section VII: 245 agents, 27 verified findings on testaspnet.vulnweb.com, 3.55h) predates instrumentation: its artifacts contain no CVSS vectors, tokens, or audit log, and key model parameters were not persisted. An honest negative on testphp.vulnweb.com reports zero confirmed SQLi findings. Appendix H's benchmark protocol (precision/recall, ablations, baselines) remains unexecuted. Overall: a well-structured agenda with candid self-assessment, but empirical assurance rests mainly on unit tests.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Assurance — truth of findings, authorization compliance, auditability — is a harness property enforceable in code, distinct from model capability (Sections I, IV).
- Five properties P1-P5 each admit formal models plus executable acceptance tests, making assurance claims falsifiable by construction (Section IV; Appendices B, D, E, F, G).
- Existing systems (Table I: PentestGPT, Cochise, MAPTA, PentestJudge, NeuroSploit) are largely silent or partial on P3-P5, though the coding is single-coder.
- The pinned NeuroSploit implementation passes its deterministic battery except for real gaps: an off-grant primary target was reconned (P4 partial) and anchoring is design-only (P5 partial) (Table III).
- An end-to-end archived run achieved 27 verified findings on testaspnet.vulnweb.com in 3.55h with 245 scheduled agents (Section VII), but predates instrumentation and therefore contains no P1-P5 artifacts.
Strengths
Falsifiable property design
Each of the five properties in Section IV pairs a formal model with an executable acceptance test (fabricated-receipt test T-P1 in Appendix B, four typed outcomes in Appendix D, CVSS cap table in Appendix E, ns-cap.v1 token spec in Appendix F, RFC 6962 anchoring in Appendix G), converting assurance from aspiration into testable code contracts.
Exemplary candor in self-testing
Table III documents the harness's own failure — an off-grant primary target was reconned and logged allow at engagement start — and Section VII reports zero confirmed SQLi findings on testphp.vulnweb.com across 245-agent schedules; the author explicitly labels the P4 gap the first fix the reference implementation owes.
Reproducible engineering artifact
Pre-registered evaluation plan
Threat model maps cleanly to properties
Weaknesses
Flagship run cannot evidence the thesis
Self-coded comparative evidence
No executed benchmark
Reference implementation itself partial
External validity thin
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper tackles a real gap: autonomous LLM penetration-testing agents (Table I codes PentestGPT, Cochise, MAPTA, PentestJudge, NeuroSploit) are judged by capability outcomes — flag capture, PoC reproduction — while assurance, meaning that findings are true (P1), reached non-destructively (P2), correctly graded (P3), within authorization (P4), and auditable (P5), goes unmeasured. The central claim is that these are harness properties enforceable in code. Three research questions structure the work: define the properties (RQ1), position existing systems (RQ2), and demonstrate realizability with cost and an evaluation plan (RQ3). The threat model (Section III) covers model hallucination (T1), hostile-target prompt injection (T2, explicitly reduced not eliminated), scope drift (T3), and after-the-fact dispute (T4, distinguishing local from full-rewrite adversaries).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does the 27-finding run on testaspnet.vulnweb.com validate the assurance framework?
No. The run predates instrumentation: all 27 findings have empty cvss fields, and no capability token or audit log exists for the run; parameters such as model, temperature, and seed were not persisted. It demonstrates capability only — assurance currently rests on the 312 unit tests and the Table III acceptance battery.
Is code and data available for reproduction?
Partially. The NeuroSploit harness is open source at pinned commit 8894649 (v4.0.0-32-g8894649, 2026-09-18), and the bundle assurance-artifacts-v1.tar.gz (40 files, 84 KB, MIT, SHA-256 root 5e018a1e...bce3) is published — though the hash is a content identifier, not an external anchor. Deterministic tests re-run without LLM spend; the LLM run statistics do not reproduce cheaply.
What did the deterministic P4/P5 battery find?
Token verification correctly rejects tampered, wrong-key, and expired tokens, and off-grant hosts are dropped from scope. But an off-grant primary target was reconned and logged allow at engagement start, so P4 scored partial; P5 scored partial because external anchoring (CT RFC 6962) is design-only while the hash-chained audit.jsonl passes its unit tests.
Are there any benchmark results?
No. Appendix H defines the evaluation protocol — 3-5 pinned vulnerable targets, at least 3 runs each, precision/recall/cost, ablations, and baselines including a conventional scanner — but the paper states it is yet to be executed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures were available only as caption fragments and equations are OCR-mangled; formal models (e.g., Eq (6)) were assessed from surrounding prose.
- The repository and artifact bundle were not independently inspected in this review; test counts and battery outcomes rest on the paper's own Tables III and IV.
- arXiv identifier, date, and authorship metadata are taken as given from the provided text.
- Whether the 27 verified findings reflect memorization of public vulnweb targets cannot be assessed from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.