PentestChain: A Cost-Aware, MCP-Orchestrated Framework for Automated Penetration Testing with Free-Tier LLMs
Rushabh Vipulkumar Patel, Dipo Dunsin, Mohammed Almaiah, Mohamed Chahine Ghanem, Mohamed Chahine Ghanem
cs.CR · cs.AI · cs.NI
Quick verdict
Free-tier-LLM pentest framework with deterministic backbone: $0 cost is measured on legacy targets only; all head-to-head and ablation evidence is projected.
Summary
PentestChain is a ten-phase automated penetration-testing pipeline (Nmap, Metasploit, Nikto, sqlmap, WPScan in a Kali container) that confines an LLM to nine bounded roles behind a cost-aware cascade — local Ollama qwen2.5:7b, then free-tier OpenRouter/Cerebras, then a deterministic fallback — exposed via an MCP server with eleven tools. Offensive logic lives in a 41-entry (port, banner-keyword) → Metasploit-module map, so exploitation proceeds even when all models fail. The claimed contributions are: measured zero-dollar cost per engagement (honestly framed as a configuration invariant), a four-position MCP threat model with four mitigations grounded in 2025 incidents (CVE-2025-6514, postmark-mcp, tool poisoning), and a containerized evaluation protocol on AutoPenBench, a Cybench subset and the PentestGPT benchmark with multi-trial statistics and same-testbed baseline reproduction. Measured results cover ten runs on five self-owned targets (Metasploitable 2, OWASP BWA, Windows 7, bWAPP, one Debian host): 26 services, 34 CVEs, ~11.5 ready exploits, confirmed shell/Meterpreter sessions, 21.7±4.4-minute pipelines, 3.6% fallback rate, $0 paid-API spend. The main limitation: every standardized-testbed, baseline-comparison and ablation result is explicitly marked 'Projected' (Table 2) — including the map-vs-model ablation that would substantiate the core claim that the deterministic backbone makes a 7B model sufficient — so 'comparable effectiveness to premium models' remains untested.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A deterministic 41-entry exploit-map backbone that confines the LLM to nine bounded roles lets a local 7B model sustain end-to-end automated penetration testing at zero measured paid-API cost.
- Dollar cost per engagement is a measured first-class metric, and $0 is a structural invariant of the local-plus-free-tier-plus-fallback configuration rather than a favourable sample.
- Exposing an offensive engine over MCP introduces a four-position threat model (malicious client, poisoned tools, client RCE, MITM/confused deputy) addressable by four default-on controls, with client-side RCE as irreducible residual risk.
- The exploit map generalised unmodified across two operating systems and four vulnerability profiles on five self-owned targets, with confirmed exploitation on three of five platforms including EternalBlue on Windows 7.
- A reproducible containerised protocol on AutoPenBench, a Cybench subset and the PentestGPT 182-sub-task benchmark with multi-trial statistics will settle the capability claims — currently projected, not collected (Table 2).
Strengths
Cost is instrumented, not asserted
Every AI call is recorded (provider, model, tokens, latency, outcome) in the ai_calls table (§5.3); Equation 1 defines paid cost over measured token counts, and §6.3 honestly frames $0 as a configuration invariant rather than a favourable measurement.
Unusually honest measured-vs-projected separation
Table 2 explicitly labels the standardized testbeds, baseline reproductions and ablations as 'Projected' and warns they 'must not be read as measured outcomes' (§6); §7 scopes capability claims as projected and reports cost as the settled contribution.
Total-function cascade design
Threat model grounded in real incidents with residual honesty
Informative negative result on a non-legacy target
Weaknesses
Core attribution claim is untested
'Comparable effectiveness' claimed without any head-to-head measurement
Thin, legacy-skewed evidence base
Statistical reporting anomalies
Enrichment false positives unexamined; novelty overstated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PentestChain and does it really cost $0 to run?
PentestChain is a research framework that automates penetration testing (reconnaissance through reporting using Nmap, Metasploit, Nikto, sqlmap and WPScan) while replacing paid LLM calls with a cascade of a local 7B model (Ollama qwen2.5:7b), free-tier OpenRouter/Cerebras APIs, and a rule-based fallback. Measured paid-API spend was $0 across ten lab runs. The zero figure holds by configuration (local and free tiers only) and excludes hardware/electricity costs and the maintenance effort for its 41-entry exploit map.
Did PentestChain beat GPT-4-based tools like PentestGPT?
No head-to-head comparison has been completed. The paper promises to reproduce PentestGPT and PentestAgent on identical testbeds, but its own Table 2 marks those results as 'projected'. All measured results come from the authors' own deliberately vulnerable lab targets (Metasploitable 2, OWASP BWA, Windows 7, bWAPP) plus one Debian host, so claims of parity with premium-model frameworks are currently untested.
Is the PentestChain code available?
The paper's Open Science appendix points to a GitHub repository (Rushabh0508/Pentest-Chain) said to contain the engine source, Dockerfile, docker-compose.yml, pinned container-image digests, prompt templates, fixed seeds and exported ai_calls cost tables, from which the measured tables can be regenerated. A DOI-archived non-anonymous release is promised with the camera-ready version.
What are the security risks of exposing a pentest engine via MCP?
The paper's threat model covers four adversary positions: a malicious MCP client redirecting scans at unauthorised targets; poisoned or rug-pulled tool descriptions injecting instructions; client-side remote code execution (exemplified by CVE-2025-6514 in mcp-remote); and man-in-the-middle/confused-deputy replay. Its default mitigations are localhost-only binding, a start-time target allowlist, exclusion of file-write/execute tools from the MCP surface, and audit logging. The client-side RCE position remains an acknowledged, engine-unclosable residual risk.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Text extraction interleaves some table and section content (Table 1 rows, overlap between §6.13 and §6.15); Figures 1–2 are not analyzable from the text.
- The abstract is truncated mid-sentence and the cost-model formulas in §6.3 are garbled in extraction.
- The identical OpenRouter/Cerebras statistics flagged in Table 6 could conceivably be an extraction artifact rather than a paper error; verify against the PDF.
- Contents of the referenced artifact repository could not be verified from the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.