SkillAtlas: An Attack Trace Library for Agent Skills
Yuxin Tian, Zenghao Duan, Liang Pang, Zhiyi Yin, Xueqi Cheng
cs.CR · cs.AI
Quick verdict
Public 3,014-case agent-skill attack-trace library; guards trained on traces hit 0.770 acc vs 0.450–0.510 baselines, but provenance and stats need scrutiny.
Summary
The paper introduces SkillAtlas, a hosted public library of attack traces for LLM-agent skills, motivated by the observation that agent-skill risk emerges from model decisions, user context, tool calls, and execution feedback rather than from static signatures or a single sandbox run. The system ingests private report bundles (primarily from the authors' SkillAttack red-teaming pipeline), parses them into case–trace–step records, applies deterministic redaction with human review, and publishes searchable public cases. The corpus spans 3,014 cases, 6,589 traces, 151,131 steps, 233 skills, 18 execution models, and 8 risk categories, dominated by Data Exfiltration (51.4%) and Bias/Manipulation (24.4%). Empirical analyses report that 724 of 1,705 successful cases (42.5%) first succeed only after a non-success initial round, and that shell/process and file evidence appears in 74.1% and 61.8% of cases. For downstream reuse, GRPO-trained guards on Qwen3-4B and Llama-3.1-8B reach 0.725 and 0.770 accuracy on a 240-example SkillRiskBench test split versus 0.450–0.510 for untrained baselines. Limitations include corpus provenance tied to a single attack framework, binary review, small-sample trigger rates without uncertainty, no statistical tests or label-verification ablation for the guard gains, and unquantified redaction efficacy.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Agent-skill risk is trajectory-level: decisive evidence often lies in intermediate tool results, errors, and judge decisions rather than final responses or static skill content.
- SkillAtlas converts private report bundles into a reviewed, redacted, searchable public library of 3,014 cases, 6,589 traces, and 151,131 steps over 233 skills and 8 risk categories.
- 42.5% of successful cases (724 of 1,705) first succeed only after a non-success initial round, demonstrating feedback-driven path dependence in agent-skill risk.
- Trajectory-verified labels plus GRPO lift pre-execution guard accuracy from 0.450–0.510 (baselines) to 0.725 (Qwen3-4B) and 0.770 (Llama-3.1-8B) on a 240-example SkillRiskBench test split.
Strengths
Genuine lifecycle gap, well argued
§1 and Appendix A.2 contrast aggregate-score benchmarks (AgentDojo, ASB, ToolEmu) with reusable trajectory evidence; the case–trace–step unit that retains failed and technical rounds is a real representational advance over final-label records.
Large, internally consistent corpus
3,014 cases / 6,589 traces / 151,131 steps over 233 skills and 18 models; derived figures all recompute exactly: 2.19 traces/case, 22.9 steps/trace, 42.5% (724/1,705), and outcome shares 56.6%/16.0%/27.4% (1,705/483/826 of 3,014).
Actionable empirical finding
Demonstrated downstream utility
Responsible-release design
Weaknesses
Single-pipeline provenance
No uncertainty anywhere
Guard ablation missing
Redaction and reproducibility unquantified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SkillAtlas (arXiv:2609.13353v2) is a system-plus-resource paper from the State Key Laboratory of AI Safety (ICT, CAS). Its thesis: agent-skill risk is *trajectory-level* — the same skill may be benign for one task and unsafe for another, and failed rounds leak dependencies and interface details that enable later success (§1, Fig. 1). Three claims: (1) trajectory evidence is a reusable representation that final labels miss; (2) a governed workflow can turn heterogeneous private report bundles into reviewed, redacted, searchable public cases; (3) these trajectories support downstream defense — retrieval, regression testing, and guard supervision. The deliverable is a hosted library with 3,014 cases, 6,589 traces, 151,131 steps, 233 affected skills, 18 execution models, and 8 risk categories.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SkillAtlas?
SkillAtlas is a hosted public library of attack traces for LLM-agent skills (skillatlas.top). It converts private security report bundles into reviewed, redacted, searchable public cases: 3,014 cases, 6,589 traces (attack attempts), 151,131 steps, spanning 233 affected skills, 18 execution models, and 8 risk categories. Raw artifacts stay private; only sanitized trajectories are published.
What is the paper's headline finding?
Two results stand out. First, 42.5% of successful attack cases (724 of 1,705) only succeed after a failed or non-success first attempt, showing agent-skill risk is feedback-driven and invisible to one-shot static or sandbox analysis. Second, guard models trained with GRPO on trajectory-verified labels reach 0.770 accuracy (Llama-3.1-8B) at predicting skill risk before execution, versus 0.450–0.510 for untrained baselines.
Is the code or dataset available?
The library itself is publicly searchable via the hosted web UI, but the paper does not state a bulk dataset dump, license, or code release for the parser or guard training, and no training hyperparameters are published. The parser also currently supports only SkillAttack export formats.
How does SkillAtlas differ from AgentDojo or Agent Security Bench?
Those benchmarks provide controlled evaluation tasks and report final labels or aggregate scores. SkillAtlas instead preserves multi-round execution trajectories — including failed rounds, tool calls, errors, and judge decisions — as reusable public evidence for search, regression testing, and training downstream guard models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 and the Figure 2 UI screenshots are images; their embedded data (e.g., Figure 3's per-category outcome shares, Figure 4's median-error-step line) cannot be read from the extracted text, so category-level outcome commentary relies on the surrounding prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.