On the Effectiveness of Kernel-Level Evidence for Agent Security
Spencer King, Zhilu Zhang, Mikhail Kuznetsov, Kay Liu, Baris Coskun, Wei Ding
cs.CR · cs.AI
Quick verdict
First paired kernel+app agent-attack corpus (ACE, 4,047 sessions); kernel evidence discriminative, fusion wins — convincing modulo label circularity.
Summary
The paper introduces ACE, a corpus of 4,047 paired agent sessions — each with an in-container strace syscall trace, the served MCP tools/list manifest, and the full agent transcript — spanning 17 threat models, six delivery-vector families, and 12 sensor-observable attack mechanics mapped to OWASP categories. Using an 86-feature featurizer (57 kernel, 29 application), the authors benchmark ten detectors across four families (boosted trees, tabular foundation models, frozen Qwen LLM judges, LoRA-tuned SLMs) under in-distribution and OWASP-aligned held-out-family regimes. Kernel evidence alone is discriminative; the concatenated Cross-View beats the application view for all ten detectors and clears a 0.75 AUROC threshold on all 12 mechanics, reaching 0.850 (XGBoost) and 0.874 (TabPFN) out-of-distribution, while the Falco rule baseline sits near chance (0.565–0.572). Three of four detector families transfer to an alternate runtime (ACE-XA). Main limitations: labels come from fire predicates that read the same signature space the detectors see, the network fingerprint reflects a sinkholed lab environment, all main captures use one agent runtime, and the corpus is only promised for release upon acceptance.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ACE is the first paired-session corpus capturing kernel syscall traces, served MCP manifests, and agent transcripts together (4,047 sessions, 17 threat models, 12 attack mechanics).
- Kernel-only evidence is discriminative for agent attacks: all four detector families exceed 0.80 OOD AUROC using kernel evidence.
- Cross-View (kernel+app) composition outperforms the App-View for all 10 detectors with 95% bootstrap CIs above zero, and is the only view clearing α=0.75 on all 12 mechanics.
- Detectors trained on ACE generalize to held-out OWASP attack families and transfer to an alternate runtime (ACE-XA) for three of four detector families.
- Off-the-shelf host-IDS rules (Falco default pack) are near-random on interactive agent workloads (0.565 ID / 0.572 OOD).
Strengths
First genuinely paired corpus
No prior agent-security benchmark or host-IDS corpus releases kernel+application evidence from the same sessions (Appendix A.1–A.4); ACE's 4,047 sessions, 17 threat models, and 12 mechanics with matched benign workloads fill a documented evaluation gap.
Attribution rigor
§2.4 argues per-session attribution structurally from PID-namespace-bounded descent; Appendix E.8 validates it empirically (30 concurrent sessions, 8,240 events, zero cross-session leaks, 100% precision) with an independent cgroup cross-check.
Statistical hygiene
Ablation depth
Candor about limits
Weaknesses
Label–feature circularity
Ecological validity
Labeling-policy dependence
No operating points
Artifacts unreleased
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a measurement question: does kernel-level (syscall) telemetry carry discriminative signal for LLM-agent attacks, and does combining it with application-layer evidence (served MCP manifest + agent transcript) beat either view alone? To answer it, the authors build ACE, a paired-session corpus of 4,047 sessions, 17 threat models, 6 delivery-vector families, and 12 sensor-observable attack mechanics mapped onto 14 of 25 OWASP LLM + Agentic categories (Table 1), plus ACE-XA, a 592-session transfer corpus on a Devstral-based ReAct runtime. Central claims: (1) ACE is the first paired kernel+application agent-security corpus; (2) kernel evidence is discriminative alone; (3) Cross-View composition generally wins; (4) detectors generalize to held-out OWASP families and transfer across runtimes. Claims are scoped honestly to post-hoc, single-session audit.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ACE dataset for agent security?
ACE (Agent Cross-Layer Evidence) is a research corpus of 4,047 recorded sessions in which an LLM agent works inside a Docker container; each session pairs a kernel-level syscall trace (captured with strace) with application-layer evidence (the served MCP tool manifest and the agent's full transcript). Part of the sessions contain hand-authored attacks across 17 threat models, so researchers can train and compare detectors that see one or both evidence layers.
Can Linux syscalls reveal when an AI agent is under attack?
According to this paper, partially yes. On their corpus, detectors reading only syscall counts and cardinalities reach roughly 0.73–0.92 AUROC on unseen attack families, and combining kernel with application evidence beats either alone for all ten tested detectors. However, pure content manipulation (text-only prompt changes) leaves no syscall footprint, and the lab network was sinkholed, so network-related fingerprints may differ in production.
Is the ACE benchmark publicly available?
Not at the time of this preprint. The authors state they will release the corpus, featurizer, detector weights, LoRA adapters, prompt templates, and per-fold results upon paper acceptance; full reproduction is estimated at roughly $1,500–$8,500 of cloud compute.
What does Cross-View detection mean in this paper?
It is feature-level concatenation: 29 application-layer features (prompt structure, tool-schema shape, tool-call trajectory) joined with 57 kernel-layer features (syscall counts, ratios, IP/path cardinalities) into one 86-dimensional vector per session. Cross-View is the only view clearing a 0.75 AUROC bar on all 12 attack mechanics and the best view for 11 of 12.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction omitted portions of §3.3 and Appendix H (per-cell Tables 8–9) and parts of Appendices K.2/L/M; some per-cell SEs were not visible.
- Figures appear only as loose numeric dumps; error bars and panel layout could not be checked directly.
- Artifact release is promised 'upon acceptance'; availability and reproducibility claims are not yet verifiable.
- Assessment of OOD magnitudes relies on prose-stated numbers (0.850, 0.874, 0.810, 0.922) since the full result tables were truncated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.