A Cyber Range Evaluation of Autonomous Network Incident Response Agents
Jakob Nyberg, Teodor Sommestad, Andrei Buhaiu, Joakim Loxdal, Pontus Johnson, Mathias Ekstedt
cs.CR · cs.AI
Quick verdict
Claims RL agents beat a heuristic in a 37-VM cyber range; credible sim-to-real study, but stats are thin and NoOp wins under Guided attacker.
Summary
The paper evaluates reinforcement-learning (RL) agents for automated network intrusion response in Crate, a cyber range built for training human operators, rather than in simulation alone. A 37-virtual-machine network (ADS-24) is attacked for two-hour episodes by the red-team emulation tool Lore under two configurations (Guided, Exploratory), while simulated user agents generate benign activity. Defender agents observe Wazuh SIEM alerts mapped to attack steps in a MAL (CadsLang) system model and may shut down or network-isolate hosts at a cost. Agents include a GNN-based RL policy trained in the MAL Simulator (Vejde), a variant trained with 1% alert noise, a heuristic rule-based agent, and a do-nothing baseline. Across 134 episodes (~30 per agent), the noise-trained RL agent achieved the best overall mean return by spending less on defenses, but the do-nothing baseline scored best against the Guided attacker under the availability-weighted cost function, and simulated-user false alerts degraded all agents, cutting returns by up to ~109%. The authors conclude RL shows promise but remains far from practical, mainly due to false-alert sensitivity. Limitations include a single untuned heuristic baseline, no significance tests for agent comparisons, and evaluation-time defects (an unmonitored attacker entry host, one unmatched attack step, silently failing defensive actions).
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A GNN-based RL policy trained in the MAL Simulator with 1% injected alert noise achieved the lowest overall mean joint attack-plus-defense cost across 134 episodes in the ADS-24 cyber range.
- The noise-trained RL agent consistently outperformed a hand-crafted heuristic that responds to every observed alert, mainly by taking fewer, better-targeted defensive actions.
- Agent performance is dominated by the adversary policy (Guided vs Exploratory) and, secondarily, by simulated-user false alerts, which reduced returns by up to ~109% under Guided.
- Under the availability-weighted cost function, doing nothing is the best policy against the Guided attacker; active defense only becomes worthwhile if attack costs are ~30% higher (or ~30% lower against Exploratory).
- This constitutes, to the authors' knowledge, the largest cyber-range evaluation of autonomous defender agents in terms of network size, time frame and attacker realism.
Strengths
Genuine sim-to-real evaluation at unusual scale
RL policies trained purely in the MAL Simulator were deployed zero-shot against a live 37-VM Crate cyber range with a production SIEM (Wazuh, incl. 586 Sigma rules) and an automated red-team tool (Lore) for month-long, 134-episode experimentation; the authors state this exceeds the scale of related range-based evaluations they could find (Sec. VII).
Transparent artifact release
CadsLang language and system model, the Vejde-MALSimulator bridge, the Wazuh-to-MAL event interface, and all evaluation data (Wazuh events, Lore ground-truth logs, user-agent logs) are publicly archived with DOIs (Secs. I, VI-C); the Wazuh-MAL interface even runs offline on saved data.
Honest reporting of unfavorable and confounding results
Robustness-oriented experimental design
Generalization-oriented agent architecture
Weaknesses
No inferential statistics behind the headline ranking
Single, deliberately naive baseline
Evaluation-environment defects discovered only post hoc
Coarse cost attribution and simulator fidelity gaps
Narrow adversary coverage
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper tests whether reinforcement-learning (RL) defender agents, trained only inside the MAL attack simulator, can outperform a hand-crafted heuristic when deployed in a physical cyber range: Crate (FOI), hosting the 37-VM ADS-24 network (CLIENT/DMZ/SRV/SOC subnets), attacked for two-hour episodes by the red-team tool Lore under two configurations (Guided, Exploratory) with optional simulated users. Agents observe Wazuh SIEM alerts - defaults plus 586 Sigma rules - mapped onto 11 attack steps of a CadsLang (MAL) instance model built from Osquery inventory, and choose between powering off a host (full availability cost) or restricting its inter-subnet connections (half cost), scored by CIA-weighted costs (Eq. 1). Central claims: (1) the noise-trained RL agent achieves the lowest overall mean joint cost and consistently beats the heuristic; (2) performance depends strongly on adversary policy and user-noise; (3) this is the largest range-based defender-agent evaluation the authors could find.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did reinforcement learning actually beat the rule-based defender in this study?
On the paper's joint attack-plus-defense cost, yes: the RL agent trained with 1% alert noise had the best overall average return and beat the heuristic agent in all four attacker/user combinations. The result is conditional, though - under the availability-weighted cost function against the Guided attacker, doing nothing at all (NoOp) achieved the best return of every agent, because any defensive action costs availability.
What are Crate and ADS-24?
Crate is a cyber range platform from the Swedish Defence Research Agency (FOI), used since 2008 to train human cyber-defense operators. ADS-24 is the 37-virtual-machine network the authors deployed inside it: a fictional maintenance-management system with four subnets (CLIENT, DMZ, SRV, SOC), Windows and Linux hosts, a Wazuh SIEM, and scripted user activity, attacked by the automated red-team tool Lore.
Is the code or data from this paper publicly available?
Partially, yes. The CadsLang modeling language and system model, the MAL Simulator-Vejde training bridge, the Wazuh-to-MAL event-mapping interface, and all evaluation data (Wazuh events, Lore activity logs, simulated-user logs) are publicly archived with DOIs. The cyber range itself and the Lore attack tool cannot be shared, so the exact experiments cannot be independently re-run.
What is the main practical takeaway for autonomous cyber defense?
Sim-to-real transfer of RL defenders is feasible - agents trained purely in simulation ran stably for a month on a real emulated network - but sensitivity to false alerts is the core blocker: benign user activity triggering SIEM alerts caused costly defensive shutdowns, cutting returns by up to about 109%, and whether an active defender beats doing nothing at all depends on the attacker's aggression and how an organization weights availability versus security.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction truncated parts of Section II-D, mid-Section V-A, and the reference list; conclusions rest on the fully legible core (Secs. III-VI, Table I, Appendices A-E).
- Figures are present only as raw text dumps, so distribution shapes (Fig. 6), ANOVA detail, and violin-plot medians could not be independently assessed.
- ANVOVA test statistics, p-values and effect sizes are absent from the extracted text; absence in the paper itself is assumed but not fully verifiable.
- The stated timestep of '1/30 second frequency' (Sec. IV) looks like a possible typo for 30 seconds; treated as unverified and avoided in judgments.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.