Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents
Joas Antonio dos Santos Barbosa
cs.CR · cs.AI · cs.SE
Quick verdict
A calibrated System One layer for LLM pentest harnesses: thoughtful architecture, candid limits, but a single-run case study gives no statistical evidence.
Summary
The paper proposes inserting lightweight, non-generative 'System One' classifiers — TypeSafe's proprietary Jev family and the open-source Laya — at four decision points of NeuroSploit, an autonomous LLM-driven penetration-testing harness: finding adjudication, severity recalibration, agent pruning, and confirmation loops (DP1–DP4). These models return typed, calibrated probabilities (Choice/Score/Noul primitives) intended to replace free-form LLM self-judgment, which the author argues causes inflated severity, accepted false positives, and wasted compute. An exploratory case study compares single runs with and without Jev against a deliberately vulnerable web target holding 13 seeded vulnerabilities: coverage was 10/13 versus 9/13, the Jev run finished 5m19s faster, findings were recalibrated toward demonstrated impact, and two harness bugs surfaced. A post-fix re-test, explicitly not a controlled pair because the harness version changed, shows a more graduated severity distribution across 22 findings. The paper further surveys the Jev/Laya landscape from vendor documentation without cross-benchmark extrapolation, analyzes RLHF/RLAIF/RLCD/RLHV training paradigms, and sketches 'Rave', a domain-adapted model, as future work. Main limitations, largely self-acknowledged: one run per condition on one target, no statistical power, potential target memorization, unquantified validator reliability, and one placeholder reference left unverified.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLM self-adjudication in pentest harnesses causes calibration collapse, class-driven severity inflation, and compute waste, which typed calibrated classifiers can fix at four formalized decision points (DP1-DP4).
- A single-run case study showed the Jev-integrated harness produced a more graduated severity distribution, ran 5m19s faster, and exposed two harness bugs — explicitly presented as non-statistical motivation.
- Vendor-reported Jev/Laya specifications suggest trade-offs in latency, calibration, and high-cardinality accuracy, but these do not transfer to pentest tasks without paired evaluation, which the authors did not perform.
- RLHV — training against the harness's own deterministic validators with proper scoring rules — yields a closed-loop, self-improving decision model whose circularity is handled by validator correctness-by-construction.
- A domain-adapted System One model ('Rave') fine-tuned via RLHV on offensive-security decision data is feasible future work with projected but unvalidated impact.
Strengths
Exceptional evidential candor
Table I pre-labels every contribution's status ('1 run/condition, 1 target'; 'Reported (not paired)'; 'Design only'); Sections IX.A, XV.A and VIII.E repeatedly mark results as non-statistical or purely hypothetical — a standard above most arXiv security preprints.
Sound decision-theoretic core
Section VI derives decision thresholds from asymmetric-loss Bayes classification (Eq. 2-3), value of information, contextual bandits with Thompson-sampling priors, and SPRT for the confirmation loop (Eq. 5), giving the threshold-based automation story of VIII.C a principled basis.
Careful additive-safety analysis
The BOLA walkthrough (IX.E) is real signal
Complete integration economics
Weaknesses
Zero statistical power by design
Confounded re-test undermines the flagship result
Reference and sourcing hygiene failures
Validator correctness asserted, never measured
No task-relevant baseline
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper is an architecture/position paper with an exploratory case study. It argues that autonomous LLM pentest harnesses err when the generative model judges its own output — calibration collapse, class-driven severity, compute waste (Sec. I) — and proposes inserting non-generative 'System One' classifiers (TypeSafe's proprietary Jev; the open-source Laya) at four decision points in NeuroSploit: DP1 finding adjudication, DP2 severity recalibration, DP3 agent pruning, DP4 confirmation loops (Sec. V). Five contributions are declared (Abstract), and Table I unusually pre-labels each as Implemented / Measured (1 run, 1 target) / Reported (not paired) / Proposed / Future work. The claims are correspondingly modest: the case study 'motivate[s] the architecture but do[es] not establish statistical significance.'
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a 'System One' model in this paper?
A lightweight, non-generative classifier that answers typed questions (Choice among options, Score on an ordered scale, Noul binary probability) in a single forward pass and returns calibrated probabilities. Here it acts as a fast 'judge' replacing LLM self-evaluation inside a penetration-testing harness. The name borrows from Kahneman's fast/slow cognition and is used by TypeSafe's proprietary Jev and the open-source Laya project.
Did adding Jev actually improve the pentest agent?
Unknown. The evidence is one run per condition on one deliberately vulnerable app with 13 seeded flaws: coverage 10/13 vs 9/13, about 5 minutes faster, and a different severity distribution. The author explicitly states this is exploratory and not statistically meaningful; a follow-up re-test changed the harness code, so it cannot isolate Jev's effect.
Is the code available?
The NeuroSploit harness is described as open-source on GitHub and Laya is Apache-2.0, but the case study depends on the proprietary TypeSafe Jev API and a subscription LLM (claude-opus-4-8). No run traces, seeds, or per-finding logs are published, so exact replication is not currently possible.
What is Rave?
A future-work proposal, not an implementation: fine-tune the open-source Laya model on offensive-security decision data using RLHV, a training loop where the harness's own deterministic validators provide ground-truth verdicts and the model minimizes Brier score against them. No results exist yet.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are available only as embedded caption text; the pipeline and training diagrams could not be visually assessed.
- Some tables (I, II) extracted with glued/garbled formatting; values were cross-checked against surrounding prose where possible.
- Vendor-reported specifications (Jev/Laya ECE, latency, benchmark accuracies) could not be verified against primary sources within this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.