SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment
Qinghua Mao, Wanying Qu, Dadi Guo, Leitao Yuan, Qingyu Liu, Yu Li, Guanxu Chen, Yanwei Fu, et al.
cs.AI · cs.CR
Quick verdict
Claims harness-policy co-evolution cuts agent attack success ~3x; results are broad and well-ablated, but single-run numbers lack significance testing.
Summary
SafeEvolve is a framework for LLM-agent safety alignment that alternates between evolving the agent's external harness (a safety system prompt and a hierarchical skill bank of general, task-specific, and common-mistake skills) and updating the policy. Trajectory-level evidence from rollouts drives bounded, versioned, rollback-able harness edits accepted by a paired accept-reject gate; a harness-use SFT cold start then teaches the policy to consume evolved artifacts, and GRPO-style RL with verifier-decomposed safety-utility rewards internalizes safety behavior. On Qwen3.5-4B, the method reports AgentDojo ASR falling from 2.37% to 0.79% with utility rising from 59.79% to 61.86%, and AgentHarm harmful score falling from 56.45 to 12.27 with refusal rising from 28.98% to 83.83%, though the benign AgentHarm score drops from 83.09 to 71.31. Similar gains are reported for Qwen3-4B-Instruct-2507, and held-out ASB evaluation shows reduced ASR across three attack families while policy-only GRPO degrades sharply on PoT Backdoor. Ablations isolate harness-only, policy-only, retrieval, proposer, transfer, and online-update effects. Limitations: all results appear to be single runs without significance tests or variance; AgentDyn utilities are small-sample quantized values; the training environment is LLM-generated simulators with self-authored rule-based verifiers, and evolution panels may overlap benchmark domains; the benign-utility cost on AgentHarm and reliance on proprietary proposer models (GPT-5.5) temper the headline tradeoff claims.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Coupling harness evolution with policy optimization achieves a better safety-utility tradeoff than harness-only or policy-only alignment (§1, Fig. 1, Fig. 3).
- Trajectory-level safety evidence can be compiled into bounded, versioned, reversible harness edits that improve safety with a frozen policy (§3.2, Tab. 2).
- Two-stage harness-use SFT plus harness-augmented RL internalizes evolved guidance, cutting Qwen3.5-4B AgentDojo ASR from 2.37% to 0.79% while raising utility (Tab. 1).
- Evolved harnesses transfer across proposers and target policies, with gains modulated by target model scale (§4.5, Figs. 5-6).
- Safety gains generalize to held-out ASB attack families where policy-only GRPO degrades severely (Tab. 8).
Strengths
Unusually complete decomposition of where the gains come from
Tab. 2 isolates frozen-policy harness effects (evolved skills: AgentDojo ASR 2.37→0.92, AgentHarm harmful 56.45→16.80), Fig. 3 contrasts model-only RL (U-Attack collapsing 60.04→26.48) with co-evolution, Tab. 5 shows prompt+skills are non-additive, Tab. 6 (OPSD) and Tab. 7 (online updates) quantify what survives harness removal and what online loops regress.
Held-out generalization evidence is the strongest result
Tab. 8 on Agent Security Bench reports SafeEvolve IPI ASR 0.50 and PoT-Backdoor ASR 2.50 versus base 20.00/GRPO 91.00 on PoT Backdoor — a genuine distribution-shift test where the policy-only baseline actively degrades.
Reproducibility scaffolding is above average for this area
Honest reporting of unfavorable results
Weaknesses
No statistical rigor anywhere
Small, quantized evaluation counts
Undisclosed train-eval domain proximity
Utility cost is benchmark-selectively framed
Dependence on proprietary frontier proposers
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SafeEvolve (arXiv:2609.02786; Shanghai AI Laboratory with SJTU, Fudan, HKUST, Zhejiang) targets safety alignment of tool-using LLM agents against two threat channels defined in §3.1: malicious user queries and indirect prompt injection embedded in environment observations. The central thesis is that neither harness-only nor policy-only alignment suffices (§1, Fig. 1) and that alternating harness edits with policy updates on shared trajectory evidence yields a better safety-utility frontier. Headline claims: on Qwen3.5-4B, AgentDojo ASR drops 2.37%→0.79% (the '3× reduction') with utility rising 59.79%→61.86%, and AgentHarm harmful score drops 56.45→12.27 with refusal 28.98%→83.83% (§1, Tab. 1); similar gains are claimed for Qwen3-4B-Instruct-2507 and on held-out ASB attacks (Tab. 8). A secondary claim is that evolved harnesses are auditable, reversible, and transferable across proposers and policies (§4.5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SafeEvolve?
SafeEvolve is a framework that improves the safety of LLM-based tool-using agents by co-evolving two things at once: the agent's runtime harness (a safety system prompt plus a library of safety skills) and the model policy itself. Safety evidence harvested from the agent's own executed trajectories drives bounded, versioned, rollback-able harness edits, and two-stage SFT-then-reinforcement-learning training internalizes that guidance into the model.
How much does SafeEvolve improve agent safety?
The paper reports that on Qwen3.5-4B, the AgentDojo attack success rate falls from 2.37% to 0.79% while utility rises from 59.79% to 61.86%, and the AgentHarm harmful score falls from 56.45 to 12.27 with refusal rising from 28.98% to 83.83%. Note the benign AgentHarm score drops from 83.09 to 71.31, and all figures come from single runs without statistical significance testing.
Is code or data available for SafeEvolve?
The paper lists a GitHub repository (github.com/MaoPopovich/SafeEvolve) and publishes hyperparameters, reward weights, and all prompt templates. However, the training environment consists of LLM-generated task simulators, the seed skill bank came from 12,800 internal rollout records, and harness edits require proprietary frontier models (e.g., GPT-5.5), so full replication needs substantial setup and GPU resources.
Does SafeEvolve's safety improvement hurt normal task performance?
Partially, depending on the benchmark. AgentDojo utility actually improves (59.79% to 61.86% on Qwen3.5-4B), but the benign AgentHarm score drops by about 11.8 points (83.09 to 71.31), and the paper's own failure analysis notes an increase in 'safe-but-incomplete' behavior where the agent stops cautiously before finishing tasks after skill updates.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-8 are rendered graphics whose internal data points (e.g., exact proposer trajectories in Fig. 5, transfer bars in Fig. 6) are not extractable; all numeric claims in this review come from the prose and Tables 1-8, which are legible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.