AgentLeak: Cloning Stronger LLM Agent Capabilities onto Weaker Agents Beyond Skill Stealing
Xiaoting Lyu, Yuhong Wu, Yufei Han, Shichang Liu, Liang Zhang, Bin Wang, Xiaobo Ma, Wei Wang
cs.CR · cs.AI
Quick verdict
声称弱智能体通过执行差异克隆更强的 LLM 智能体能力（>80% 的差距恢复率）；结论令人信服，但统计报告不足。
Summary
AgentLeak 是一种黑盒攻击，表明窃取 LLM 智能体的技能并不能转移其解决任务的能力。在动机研究中，基于 Qwen3.6-35B 的 OpenHands 攻击者即使拥有受害者自身的技能，其通过率也仅达到 30.64%，而受害者为 80.44%（图 1）——即所谓的“技能执行差距”。该攻击将这种差距视为泄露面：每个场景使用一个受害者执行查询，它抽象出一个工作流技能，从受害者-攻击者轨迹差异中诊断出能力缺陷，将缺陷转化为行为基元，并对攻击者侧的技能应用结构化的 Insert/Replace 补丁，迭代直到成功或预算耗尽。在扩展到 600 个 LLM 生成实例的 20 个 SkillsBench 场景中，AgentLeak 在四种受害者系统（Claude Code/Opus 4.8, Codex/GPT-5.5, OpenHands/DeepSeek-V4, OpenHands/MiniMax-M3）和两个弱攻击者的所有配置下，将攻击者通过率提升至 70% 以上，能力恢复率为 85–108%，通常超过受害者本身；重构的技能可跨攻击者模型和框架进行转移（表 5）。消融实验将最大份额归因于工作流抽象（没有它，CRR 会降至 2.00%）。主要局限：每个实例运行三次，没有置信区间或显著性检验；留出实例是共享重构工作流的 LLM 生成变体；GPT-5.6 分析助手比所有受害者都强；没有说明代码或实例发布；且算法 1 假设受害者在查询实例上成功。
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Stealing a stronger agent's skills does not transfer its task-solving capability: with identical skills a Qwen3.6-35B attacker reaches 30.64% pass rate vs the victim's 80.44% (Figure 1), a disparity the authors call the skill execution gap.
- Victim-attacker execution differences form a black-box leakage surface from which capability-critical procedural behaviors can be diagnosed and externalized into attacker-side skills.
- AgentLeak achieves >70% IPR across all eight victim-attacker configurations and recovers 85-108% of the capability gap using a single victim query instance per scenario, at ~$6.32 assistant cost and ~60 minutes per scenario.
- Reconstructed skills transfer across attacker models and harnesses without re-reconstruction, and existing artifact-level defenses (Table 7) reduce CRR by only 7-18 points while trajectory redaction reaches merely 73.23% CRR.
Strengths
Novel, clearly formalized problem
The skill/artifact vs capability leakage distinction (§2.1, §3) is new: Figure 1 shows identical skills yield 30.64% pass rate on a Qwen3.6-35B attacker vs 80.44% for the Codex/GPT-5.5 victim, cleanly motivating the 'skill execution gap' as a leakage surface.
Method specified to a reproducibility-grade standard
Algorithm 1, patch semantics (Insert/Replace with location+content), deficit/primitive record schemas, full prompt templates (Appendix A), and a worked before/after skill example (Appendix B) are all published.
Broad evaluation skeleton
Informative ablations
Honest limitations section
Weaknesses
No uncertainty quantification
Instance-construction circularity
Frontier assistant undermines the weak-attacker framing
Algorithm edge case unhandled
Clone-implies-cloning claim blurs
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AgentLeak (arXiv 2609.07131) is a security attack paper on LLM agents. Its central conceptual claim is a separation between artifact leakage and capability leakage: stealing a victim agent's skill files does not reproduce its task-solving capability, because critical procedural behaviors (decomposition, tool selection, feedback interpretation, verification) are realized only at execution time. The paper formalizes this as the "skill execution gap" — Figure 1 shows identical skills yielding 30.64% pass rate on a Qwen3.6-35B/OpenHands attacker versus 80.44% for the Codex/GPT-5.5 victim, with trajectory distillation (Trace2Skill) reaching only 34.96% — and then closes the gap adversarially: a black-box attacker reconstructs victim capability into its own skills from observable execution differences, using one victim query per scenario. Supporting claims: (1) AgentLeak exceeds 70% IPR in all eight victim-attacker configurations (Table 4); (2) reconstructed skills transfer across attacker models and harnesses (Table 5); (3) artifact-level defenses are largely ineffective (Table 7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'skill execution gap' in AgentLeak?
It is the paper's term for the finding that possessing a skill does not equal possessing the capability it enables. In their motivating test, an OpenHands agent with a Qwen3.6-35B model using the victim's own skills scored 30.64% pass rate versus 80.44% for the victim (Codex with GPT-5.5), because critical behaviors — task decomposition, tool selection, verification, error recovery — are realized at execution time rather than written into the skill file.
How much access to the victim does the AgentLeak attacker need?
Black-box only: one task instance submitted to the victim per scenario, observing its normal interface output (tool calls, observations, outcomes). The reported averages are 6.34 attacker executions, 0.32M auxiliary-model tokens, about $6.32 and 60 minutes per scenario. No access to private skills, model weights, system prompts, or hidden reasoning is assumed.
Do existing defenses stop the attack?
Poorly, according to the paper's Table 7. Skill-artifact defenses (input-phase detection, SkillGuard-5, semantic output filtering) cut the capability recovery rate from 95.44% to only 77.22-88.34%. Trajectory redaction (removing intermediate tool-call details) works best but the attack still recovers 73.23% of the gap, so the authors argue behavioral disclosure control needs new mechanisms.
Is code or data available for AgentLeak?
The paper does not state a code or dataset release. It builds on the public SkillsBench benchmark, publishes its full prompt templates in Appendix A, and describes instance construction in Appendix C, but the 600 generated task instances and the attack pipeline are not announced as released, and results depend on specific commercial model versions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Metadata author list (8 names) omits Wei Wang, who appears in the paper byline as corresponding author (Wei Wang1†) — a metadata/text discrepancy, not an extraction loss.
- Numeric labels inside Figures 1/3/6 were extracted as unstructured number strings, so figure-cited values (e.g., Figure 1: 80.44/20.13/30.64/34.96) rely on positional mapping; all table values are cleanly extracted. The text runs complete through Appendix D with no evidence of truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.