Staying on the Attack Path: Structured State for Long-Horizon Automated Penetration Testing
Weizhe Wang, Yitong Zhang, Yao Zhang, Xiaoqiang Di, Zhigang Li, Bin Wu, Guangquan Xu
cs.CR · cs.AI
Quick verdict
Claims fact-intent DAG lifts LLM pentest agent to 88.2% on CTFs vs 44.1% VulnBot; convincing but n=34, no DAG ablation, and thin Cairn attribution.
Summary
The paper presents Intentest, an LLM-based automated penetration testing agent built on the open-source Cairn framework, which externalizes long-horizon agent state into a persistent fact-intent DAG: verified network facts become immutable nodes, and exploration intents become edges valid only when grounded in predecessor facts. A three-layer architecture adds a scheduler (priority task types, 180-second timeouts with a 90-second degradation-recovery phase, adaptive load balancing) and a five-stage intent retrieval pipeline (embedding prescreen, subgraph-isomorphism verification, Jaccard fallback, goal alignment, LLM heuristic) that injects tactical priors into the executing model. On 34 real web CTF challenges (11 easy, 11 medium, 12 hard) with all agents running DeepSeek V4 Pro, Intentest reaches 88.2% overall success (100/90.9/75.0 by difficulty) versus 44.1% for the strongest baseline, VulnBot, with one-sided Fisher's exact tests reported per tier. The ablation shows the intent module cuts successful-task rounds by about 33% (medium) and 48% (hard) without changing the solvable set, attributing solvability to the DAG and scheduling. Main limitations: no controlled ablation of the DAG itself (the cross-system VulnBot comparison is corroborating only), a small single-run benchmark, no code or benchmark release, thin explanation of what is inherited from Cairn, and efficiency comparisons conditional on success.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Externalizing long-horizon state into a persistent fact-intent DAG raises overall CTF success to 88.2% (75.0% on hard) versus 44.1% (25.0%) for VulnBot, the strongest baseline, with all agents on DeepSeek V4 Pro (Table 1).
- The intent retrieval and prediction module cuts successful-task rounds by ~33% on medium and ~48% on hard tasks without changing the set of solvable tasks (Table 3).
- Graph-structured state — not orchestration alone — is the decisive factor for solvability, inferred from the noIntent ablation plus the VulnBot contrast (Sect. 6.2, acknowledged as corroborating rather than conclusive).
- DAG precondition validation rejects intents not grounded in verified facts by construction, confining indirect-prompt-injection risk to degraded decision quality rather than unauthorized execution (Sect. 4.4, 7).
Strengths
Formalized, implementable design
Definition 1 (fact-intent DAG quadruple), Definition 2 plus Eq. 1 (four-state FSM), and Algorithms 1-3 make the DAG, scheduler, and retrieval pipeline precisely reproducible in intent; precondition validation (Sect. 4.4) rejects intents not grounded in verified facts by construction, bounding injection risk.
Honest statistical reporting
Sect. 5.2 explicitly reports non-significant easy-tier gaps vs Pentest-R1 (p=0.107), VulnBot (p=0.107), and Claude Code (p=0.238) instead of hiding them; medium/hard and overall p-values are given for every baseline.
Clean attribution of roles via ablation
Above-norm failure and validity analysis
Controlled model confound
Weaknesses
Core component never ablated
Small, single-run evaluation
Unclear Cairn lineage
No artifacts released
Efficiency claims carry selection bias and an unreported metric
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper proposes Intentest, an LLM-driven automated penetration testing agent built on the open-source Cairn framework [16], targeting two failure modes of security agents on long-horizon tasks: context forgetting and intent drift. The core mechanism is a persistent fact-intent DAG (Definition 1, Sect. 4.2) in which verified network states are immutable fact nodes and exploration intents are edges that must be grounded in predecessor facts; a four-state FSM (Definition 2, Eq. 1) governs intent lifecycle. Claims: (1) 88.2% overall success on real web CTF challenges versus 44.1% for VulnBot, the strongest baseline (Table 1); (2) the intent retrieval module cuts successful-task rounds ~33%/~48% on medium/hard without changing the solvable set (Table 3); (3) graph state — not orchestration alone — is decisive for solvability (Sect. 6.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Intentest?
Intentest is an LLM-based automated penetration testing agent, built on the open-source Cairn framework, that keeps long-horizon state in a persistent fact-intent directed acyclic graph: verified network facts become immutable nodes, and each exploration intent must be grounded in predecessor facts before it can be executed.
How successful is Intentest on CTF challenges?
On 34 real web CTF challenges (11 easy, 11 medium, 12 hard), with every compared agent running the same LLM (DeepSeek V4 Pro), Intentest solved 88.2% overall and 75.0% of hard tasks, while the strongest baseline, VulnBot, reached 44.1% and 25.0% respectively.
Is Intentest's code or benchmark publicly available?
The paper does not announce a code or benchmark release. The system builds on the open-source Cairn project and reuses the public Pentest-R1 training corpus for its offline intent graph (minus blacklisted test challenges), but the 34-challenge benchmark and evaluation harness are not stated to be available.
What problem does the fact-intent DAG solve?
It targets context forgetting and intent drift in long penetration sessions: because state lives in a graph database rather than the LLM's context window, early critical facts survive across dozens of steps, and injected or hallucinated intents without verified-fact grounding are rejected by precondition validation. The paper's ablation indicates this graph structure — not the retrieval module — is what enables solving hard tasks.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract string supplied in the metadata is cut mid-sentence, but the complete abstract is present at the start of the extracted full text; no content loss.
- Figures 1, 3, and 4 appear only as embedded captions/labels in the extracted text; their graphical content (architecture diagram, exploration-path graphs) could not be visually verified, and the 11/11/12 task split is inferred from Table 1 percentages rather than stated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.