Where Cyber Agents Struggle: Bottleneck Analysis of Multi-Stage LLM Agents
Saeedeh Lohrasbi, Mohammad Mamun, Ahmed Yehia, Scott Buffett, Sherif Saad
cs.CR · cs.AI · cs.SE
Quick verdict
Careful diagnostic study of multi-stage LLM cyber agents: bottlenecks cluster in planning/execution; solid but single-run, LLM-judged, no human validation.
Summary
This paper presents a diagnostic framework for locating where multi-stage LLM-based cyber attack agents stall, going beyond binary success rates. Built on the Autonomous Adversary architecture (an acting LLM as orchestrator, a cyber executor, and an LLM validator), the study evaluates six frontier models across two enterprise-like lateral-movement scenarios and three scaffolding modes (expert-defined, self-scaffolded, fully autonomous). Analysis is two-resolution: a subtask-conditioned, cost-aware z-score localizes abnormal token use, retries, and runtime under expert decompositions, while a five-model LLM-as-a-Judge panel assigns primary trajectory bottlenecks from seven categories, plus a separate validator-evidence assessment. Across 29 complete judge panels (145 judgments), bottlenecks cluster in credential and lateral-movement tasks and shift with scaffolding: under self-scaffolding all four comparison models are planning-limited, while under expert decomposition action execution dominates. Validator explanations score 0.79–0.84 overall and are evidence-grounded (0.81–0.90) but nonspecific (about 0.68–0.69) and more cautious about failures (0.92 vs 0.70); validator/framework factors appear in 72–100% of judgments. Mean severity spans 4.03–5.00 and mean modal judge support 0.60–0.90. Limitations are substantial and acknowledged: best-trajectory selection (one run per configuration), no human ground truth for judge diagnoses, no active defender, arbitrary equal score weights, and one panel lost to a provider safety filter.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 1
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Success rates alone are inadequate for evaluating multi-stage cyber agents; reliable evaluation must also cover evidence interpretation, resource use, and adaptation after failure.
- Diagnosed bottlenecks cluster in credential-acquisition and lateral-movement tasks and shift with scaffolding mode: all four comparison models are planning-limited under self-scaffolding, while action execution dominates under expert decomposition.
- System validators are generally relevant and evidence-grounded but nonspecific and overly optimistic, with validator/framework contributions appearing in 72–100% of judgments across settings.
- Judge agreement is genuinely partial (mean modal support 0.60–0.90; 11 of 23 cohort panels lack one-vote robustness), so several diagnoses carry substantive interpretive uncertainty.
Strengths
Two-resolution diagnostic design connects where and why
The subtask-conditioned cost z-score (Eq. 1) localizes relative resource amplification even in successful subtasks, while the trajectory taxonomy explains why; §5.2 explicitly demonstrates comparable resource scores receiving different diagnoses — a real methodological contribution beyond success-rate reporting.
Unusually explicit epistemic hygiene throughout
Cohort-dependence of the score, best-trajectory selection bias, non-independence of judge counts, and the safety-filter-blocked Opus S2-A panel (excluded from all aggregates, marked INC rather than imputed) are all disclosed at point of use rather than buried.
Separates validator/framework failures from acting-model failures
Honest agreement reporting with a reproducible robustness check
Artifacts released
Weaknesses
No human ground truth for any diagnosis
Single-run, best-trajectory selection biases all reported numbers
Fragile modal support in cells driving conclusions
Localization score weights are arbitrary and untested
Limited ecological validity for offensive security claims
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a diagnostic, not a leaderboard, question: within observed executions of a multi-stage LLM cyber-attack agent, where does progress become constrained, and what persistent weakness best explains it (§1)? The system under study is the Autonomous Adversary architecture — an acting LLM (orchestrator), a cyber executor returning stdout/stderr/exit status, and an LLM validator gating the loop — evaluated in an enterprise-like range across two lateral-movement scenarios (S1: nine expert subtasks from a low-privilege workstation; S2: ten from an administrative jump server) and three scaffolding modes (expert-defined, self-scaffolded, fully autonomous). Six frontier models are covered; a four-model cohort (Opus 4.5, Sonnet 4.5, Gemini 3 Pro Preview, GPT-5.1) provides consistent cross-setting coverage (Table 1). The authors explicitly disclaim model ranking and causal claims; the contribution is a two-resolution diagnostic framework plus a case study.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the paper 'Where Cyber Agents Struggle' about?
It is a diagnostic study from Canada's National Research Council that observes LLM-powered cyber-attack agents executing multi-step intrusion scenarios and analyzes where they get stuck and why, instead of only reporting success or failure. Six frontier models are evaluated across two enterprise-like lateral-movement scenarios and three levels of task guidance (expert-defined, self-scaffolded, fully autonomous).
What are the main bottlenecks found in LLM cyber agents?
Bottlenecks concentrate in credential-acquisition and lateral-movement steps. With expert-provided task decompositions, most models fail mainly at action execution (commands, syntax, tool invocation); when models build their own plans, all four comparison models are limited by planning instead. Validators are evidence-grounded but nonspecific and overly optimistic, and validator/framework factors appear in 72–100% of judgments.
Is the code and data for the paper available?
Yes. The paper links a data repository (doi.org/10.4224/40004009) and source code on GitHub (wherecyberagentsstruggle/LLM_as_Judge). Re-running the experiments themselves requires the original enterprise-like cyber range environment and paid access to frontier LLM APIs.
Does the paper rank which LLM is best at cyber attacks?
No. The authors explicitly state the contribution is diagnostic, not a representative model ranking or causal demonstration. They analyze only the best-performing run per model–scenario–mode configuration, so the reported numbers cannot be read as average failure rates or model standings.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 are images; the heatmap values underlying §5.2 localization claims could not be inspected from the extracted text.
- Judge-panel model names (DeepSeek V4 Pro, GPT-5.6 Sol/Luna, Kimi K3, Gemini 3.6 Flash) cannot be cross-checked from the provided text alone.
- Extraction appears complete through the references, but judge prompts, per-run token/time inputs, and any appendix material are absent from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.