Skynet: Workflow-Level Anomaly Detection for Agentic AI via Semantic and Structural Modeling
Chaoyu Zhang, Hexuan Yu, Heng Jin, Shanghao Shi, Ning Zhang, Yi Shi, Yulia R. Gel, Y. Thomas Hou, et al.
cs.CR · cs.AI
Quick verdict
Benign-only workflow-level anomaly detection for agentic AI; strong 100%-recall/0.89%-FPR results but single runs, small anomaly sets, domain-shift confound.
Summary
Skynet is a workflow-level anomaly detection framework for agentic AI that converts multi-agent execution traces into directed workflow graphs and scores them against a benign-only learned model. Graph nodes are execution steps (planning, tool calls, inter-agent messages, system events); typed edges encode delegation, data flow, and temporal-causal order. Training uses contrastive learning over a dual-branch encoder (semantic plus structural) with invariant-preserving positive views and counterfactual invariant-violating negatives derived from four workflow invariants (role, provenance, dependency, stability). Scoring measures distance to the nearest of K=150 k-means benign modes, thresholded at a benign-score quantile. Trained only on benign ETO trajectories, it is evaluated on Agent-SafetyBench (111), AgentErrorBench (200), ATBench (497), and an 808-trajectory Combined set: 100% recall at 0.89% FPR on the first two, 82.70% on ATBench, 89.36% on Combined, beating six baselines in the low-FPR region, with sub-second per-workflow latency. Main limitations: benign training source and anomaly benchmarks come from different generators, so dataset-shift confounds are unexamined; results are single runs without variance estimates; full-model numbers differ between Tables 1 and 3; finest subclasses have 1-13 samples; no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Workflow-level graph representations expose non-local anomalies (role-inconsistent delegations, atypical tool-call pathways, abnormal artifact influence) that per-step inspection misses.
- Contrastive training with invariant-based counterfactual negatives, using benign data alone, yields a single decision rule covering adversarial manipulations, intrinsic failures, and zero-day anomalies.
- Skynet sustains 100% recall at 0.89% FPR on Agent-SafetyBench and AgentErrorBench and the best recall-FPR trade-off on ATBench and Combined, at one fixed operating point.
- Each workflow-aware component contributes: removing single-mode scoring collapses recall to 65.59%, and removing counterfactual views or workflow-aware augmentation pushes FPR beyond the 1% budget (Table 3).
- Per-workflow (~0.08 s mean) and per-step (4.8-48.8 ms typical) latencies are compatible with always-on online monitoring (Table 4).
Strengths
Principled, invariant-grounded view construction replacing random graph augmentation
T+/T- views are built from four documented workflow invariants (role, provenance, dependency, stability), each empirically tied to known anomaly families (Sec. 3.2); the Table 3 ablation supports the design, with removal of workflow-aware augmentation raising FPR to 0.98% and cutting recall to 93.54%.
Single fixed sub-1% FPR operating point with strong cross-scenario transfer
Skynet holds 0.89% FPR across all four scenarios; on Combined (808 anomalies) it reaches 89.36% recall where Guardian falls to 49.75% and TraceAegis to 58.42% (Table 1), with ROC curves (Fig. 4) supporting the threshold-independent comparison.
Honest reporting of baseline wins and failure modes
Clear problem formalization connecting anomaly granularity to the NIDS packet-to-flow analogy
Credible runtime characterization for the deployment claim
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Skynet (arXiv 2609.06835v2, MobiHoc '26) argues that anomaly detection for agentic AI must operate at the workflow level rather than per prompt or step, mirroring the historical shift from packet-based to flow-based network intrusion detection (Sec. 1). The system consumes execution records (ReAct traces, MCP tool calls, inter-agent messages, system events), builds directed workflow graphs, and scores them against a benign-only learned model. Central claims: (1) non-local anomalies (role-inconsistent delegation, atypical tool pathways) are visible only globally; (2) benign-only contrastive training with invariant-based counterfactuals covers adversarial manipulations, intrinsic failures, and zero-day anomalies under one rule; (3) 100% recall at 0.89% FPR on two of three benchmarks with online-compatible latency.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Skynet and what does it detect?
Skynet is an anomaly detection framework that converts multi-agent AI execution traces into directed workflow graphs and scores them against a model learned from benign executions only. It jointly encodes execution semantics and dependency structure, flagging workflows that deviate from learned benign patterns — covering prompt injections, tool misuse, and intrinsic agent failures under one rule.
How well does Skynet perform?
Per the paper, it detects 100% of anomalies at 0.89% false positive rate on Agent-SafetyBench and AgentErrorBench, 82.70% recall on ATBench, and 89.36% on an 808-trajectory combined set, beating six baselines in the low-FPR region. It reports these as single-run results on three public benchmarks.
What data was Skynet trained on?
Only benign trajectories from the ETO dataset (AlfWorld, SciWorld, WebShop environments) — 4,494 training workflows. No attack or failure data is seen during training; anomalous workflows come from three external public benchmarks used purely for evaluation.
Does Skynet really detect zero-day attacks?
The claim rests on anomalies surfacing as off-manifold geometry under one distance rule, and on benchmarks whose anomaly types differ from any training signal. However, the evaluated anomalies all come from published benchmark suites, so the zero-day claim is an extrapolation rather than a directly demonstrated result.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are images; their plotted content (ROC curves in Fig. 4, UMAP geometry in Fig. 3) could only be assessed from captions and surrounding prose, not the plots themselves.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.