Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents
Xiaofang Yang, Ziqi Miao, Dianbo Sui, Jing Shao, Lijun Li
cs.CR · cs.AI
Quick verdict
Proposes SkillSonar, a skill-native runtime guard evolved by MCTS, cutting SCOPE-R ASR ~4-6x; credible, but soft enforcement and LLM judging temper strength.
Summary
The paper argues that pre-install vetting of agent skills is insufficient because malicious skills can defer harmful behavior until runtime context makes it appear useful. It proposes Defense-as-Skill: the runtime guard itself is an installable, inspectable skill. SkillSonar loads alongside untrusted task skills in Claude Code and OpenClaw and routes each proposed action to allow, replan/narrow, or confirm against the user's task boundary. To evaluate this, the authors build SCOPE-R, with 206 attack-confirmed malicious skill instances and 43 benign tasks across six risk families and 21 sub-categories, generated by a closed-loop attacker-victim-judge pipeline over 15 PinchBench tasks. The guard is then optimized by runtime guard-skill evolution, a Monte Carlo Tree Search over on-disk guard variants scored by attack success, utility, confirmation burden, and token cost. On repeated GLM-5 runs (N=10), SkillSonar lowers ID ASR from 0.482 to 0.104 and OOD ASR from 0.606 to 0.115 while benign utility stays near no-guard levels; gains transfer to Claude Haiku 4.5, GPT-5.4, held-out risk families, two external benchmarks, and survive adaptive attackers (45.1% to 22.9%). Key limitations: protection is soft instruction-following rather than mechanism-level enforcement; results depend on an LLM judge (audited on 18-32% of samples); baselines are sparse; and the text contains internal inconsistencies (8 vs 10 full evaluations; GPU vs CPU-only claims).
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Pre-install vetting is insufficient because malicious skills defer harm until task and workspace state make unsafe actions useful (Sec 1).
- A runtime guard implemented as an ordinary skill (SkillSonar), with explicit safety-responsibility invocation, blocks skill-induced attacks without modifying the agent runtime on both Claude Code and OpenClaw (Sec 4).
- On repeated GLM-5 runs (N=10), MCTS-evolved SkillSonar cuts ID ASR from 0.482 to 0.104 and OOD ASR from 0.606 to 0.115 while benign utility stays near no-guard levels (Tables 2, 5).
- Gains transfer zero-shot to held-out risk families, other victim models, and external benchmarks, and persist under adaptive attack (45.1% to 22.9%) (Sec 5.3, Appendix F).
Strengths
Deployment-realistic, honestly framed defense
SkillSonar needs no runtime modification and runs from the same skill bundle on both Claude Code and OpenClaw (Sec 4.1); the paper openly treats soft invocation (Appendix D.1) and its non-enforcement nature as deployment facts rather than hidden caveats (Conclusion, Appendix H).
Attack-confirmed dataset with diagnostic taxonomy
Algorithm 1 retains only instances where the judge confirms the target behavior actually executed on a victim rollout (206 instances, 6 families / 21 sub-categories, Table 7); human audit (Table 8) shows all observed judge errors are false-positive successes, biasing reported ASR against — not for — the defense.
Mechanism-isolating ablations
Unusually broad evaluation surface
Weaknesses
Sparse baseline field
Inconsistent run counts and no inferential statistics
Internal inconsistencies undermine reproducibility details
Guard remains exploitable by its own core threat
Narrow attack-source diversity and weaker second-harness results
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets runtime safety of skill-augmented agents (SAAs) such as Claude Code and OpenClaw, where a loaded skill can steer tool use long after installation. It contributes: (1) SCOPE-R, a task-conditioned dataset of 206 attack-confirmed malicious skill instances plus 43 benign tasks spanning six risk families — Specification Integrity, Capability Control, Operational Side Effects, Privacy & Data Flow, Execution Safety, Resource & Reliability — and 21 sub-categories (Sec 3, Table 7); (2) SkillSonar, a runtime guard packaged as an ordinary skill that triages proposed actions against the user's task boundary into allow / narrow-replan / confirm via R0-R3 risk tiers (Sec 4.1, Appendix C.3); (3) runtime guard-skill evolution, an MCTS procedure that edits the on-disk guard from rollout feedback (Sec 4.3, Algorithm 3). Central claim: the evolved guard cuts ASR roughly 4-6x on repeated GLM-5 runs while preserving utility, transfers across models, families, and benchmarks, and survives adaptive attacks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Defense-as-Skill?
A defense paradigm for skill-augmented AI agents in which the runtime safety guard is itself packaged as an ordinary agent skill — installable, inspectable, and editable — rather than an external classifier or platform patch. The paper's instantiation is SkillSonar, which checks each proposed action against the user's task boundary and decides to allow it, narrow/replan it, or require user confirmation.
What is the SCOPE-R dataset?
A task-conditioned safety benchmark for skill-augmented agents: 206 malicious skill instances whose attacks were confirmed to actually execute against a victim agent, plus 43 benign tasks for false-positive measurement. It covers six risk families (Specification Integrity, Capability Control, Operational Side Effects, Privacy & Data Flow, Execution Safety, Resource & Reliability) with 21 sub-categories, built on 15 PinchBench tasks via an attacker-victim-judge loop.
How effective is SkillSonar, and does it replace sandboxing?
On repeated GLM-5 runs it lowers attack success rate from 0.482 to 0.104 on seen risk families and from 0.606 to 0.115 on held-out families, with benign-task utility near unguarded levels, and it retains protection under adaptive attacks (45.1% to 22.9%). However, it is a soft, instruction-following layer: the authors state it should complement — not replace — permission systems, sandboxing, and other mechanism-level controls.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal text (Figs 1-6) is garbled by PDF extraction; all quantitative values cited in this review were verified against tables or prose. Body text appears complete through Appendix H (p. 22).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.