HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution
Wen Jiang, Mingmin Chu, Yimeng Tian, Qianxin Zhang, Haofei Yang, Rui Yang, Yang Liu, Tao Lv, et al.
cs.LG · cs.AI
Quick verdict
HarnessEvolve evolves the agent harness via reference trajectories and dual gates, beating GEPA/ACE/SkillOpt on 5 benchmarks, without significance tests.
Summary
HarnessEvolve, from Huawei's ICT AI Competence Center, is a self-evolving agent framework that optimizes an agent's entire harness — prompts, skills, tools, and execution logic — rather than one component. Four decoupled LLM agents collaborate: an execution agent runs tasks; an evaluation agent grades accuracy and verifies reference trajectories (execution paths generated when the ground-truth answer is supplied); an optimization agent aligns failed trajectories against references to locate the first divergence point, clusters errors into systematic patterns, and proposes harness edits; and a gate agent enforces a quality gate (data-leakage and prompt-bloat checks) plus a performance gate that accepts an update only if it improves the current batch without degrading the last two, with epoch-end validation selecting the best snapshot. Across SearchQA, OfficeQA, SpreadsheetBench and two in-house enterprise datasets, with Qwen3.6-27B and DeepSeek-V4-Flash, the paper reports the highest accuracy in every setting, e.g., 86.9% versus 65.3% for GEPA on CloudCoreNetwork-QA; ablations attribute the largest effect to reference-guided diagnosis (86.9% to 57.8%). Limitations: no statistical testing or variance reporting; headline gains rest on non-public data; baselines optimize single components, confounding method with optimization scope; no compute-cost analysis; the performance gate itself is not ablated; and no code release is mentioned.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reference-guided error diagnosis — aligning failed trajectories with ground-truth-conditioned reference trajectories to find the first divergence point — is the most critical component, with ablation dropping CloudCoreNetwork-QA accuracy from 86.9% to 57.8%.
- Optimizing the entire harness outperforms single-component optimizers (GEPA, ACE, SkillOpt) on all five benchmarks with both backbone models.
- The quality gate filtering data leakage and prompt bloat prevents shortcut learning (ablation M3: 86.9% to 80.1%).
- The performance gate with a two-batch replay buffer and epoch-end validation prevents catastrophic forgetting while retaining beneficial updates.
- HarnessEvolve-optimized skills transfer to four unseen agent frameworks without re-optimization, improving or maintaining accuracy in every cell of Table 3.
Strengths
Structurally isolated gating
Decoupling execution, evaluation, optimization, and gating into separate agents (Section 3.1, 3.5) means the optimizer cannot grade its own updates; the gate agent returns rejection reasons and enforces a revision loop (Trev=3), an auditable design absent from prior harness-evolution work like Self-Harness and AHE as characterized in Section 2.
Mechanistic answer to credit assignment
Pre-generating verified reference trajectories by rerunning Aexec with ground-truth answers (Section 3.2) and localizing the first action divergence (Section 3.4) turns binary failure feedback into a root-cause signal; ablation M1 shows the largest effect of any component (86.9% down to 57.8% on CloudCoreNetwork-QA with Qwen3.6-27B, Table 4).
Broad experimental coverage
Full hyperparameter disclosure
Thoughtful clustering details
Weaknesses
No statistical rigor
Headline gain is unverifiable
Baseline-scope confound
Performance gate never ablated
Uncalibrated LLM-judge checks
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
HarnessEvolve (Huawei ICT AI Competence Center, arXiv:2609.00829) is a systems/method paper on self-evolving LLM agents. It targets three failure modes of naive self-evolution — credit assignment failure, shortcut learning, and catastrophic forgetting — and claims four contributions: reference-guided error diagnosis, a quality gate against shortcut learning, a performance gate against forgetting, and a decoupled multi-agent architecture that optimizes the entire harness (prompts, skills, tools, execution logic) rather than a single component. The headline empirical claim is highest accuracy on all five benchmarks (three public, two in-house) across two backbone models.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is HarnessEvolve?
HarnessEvolve is a framework from Huawei's ICT AI Competence Center that lets an LLM agent automatically improve its own harness — its prompts, skills, tools, and execution logic — by comparing failed task runs against verified reference trajectories (runs produced when the model is given the correct answer) and filtering every proposed update through a quality gate and a performance gate.
How does HarnessEvolve solve the credit assignment problem in agent self-evolution?
It pre-generates reference trajectories by running the agent with the ground-truth answers, then aligns each failed trajectory against its reference to pinpoint the first action where the two diverge; that first divergence is treated as the root cause, and the resulting error signals are clustered into systematic patterns before any harness edit is proposed.
What results does the HarnessEvolve paper report?
The paper reports the highest accuracy in every tested configuration: on public benchmarks it reaches 92.9% on SearchQA, 70.9% on OfficeQA, and 76.4% on SpreadsheetBench, and on Huawei's internal CloudCoreNetwork-QA dataset it reaches 86.9% versus 65.3% for the strongest baseline (GEPA), a 21.6 percentage-point gain.
Is HarnessEvolve's code or data publicly available?
The paper does not mention any code release, and two of its five benchmarks (CloudCoreNetwork-QA and Wireless-QA) are internal Huawei datasets that are not public, so the largest reported gains currently cannot be independently verified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are image content; only their captions entered the text extraction, so the self-evolution curves (Figure 2) and the qualitative before/after harness diffs (Figure 3) could not be inspected, and claims resting on those figures were assessed from text alone.
- The extracted Figure 1 region contains repeated code fragments (future.result/print lines) that appear to be figure-embedded text or an extraction artifact; the figure's actual layout could not be verified from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.