EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?
Zixuan Ke, Vaidehi Patil, Haizhou Shi, Yang Li, Ye Liu, Sarath Shekkizhar, Anurag Koul, Jiayu Wang, et al.
cs.MA · cs.CL
Quick verdict
针对 LLM 智能体工具演进的首个基准测试；设计扎实且具备机制分析，但启发式的技能/智能体标签和单薄的数据统计削弱了其结论的说服力。
Summary
EvoHarnessBench 是一个用于评估 LLM 智能体的基准测试，其评估场景是外部提供的工具支架（harness）——即智能体可用的工具、可复用技能和专业智能体——随时间推移而扩展，这与之前改变任务流的基准测试形成对比。作者确定性地将两个基于验证器的基准测试（EnterpriseOps-Gym 和 Agentic Last Exam）转换为 17 个 3-6 阶段的累积工具支架流，涵盖 802 个任务、520 个工具、42 个技能和 62 个智能体；每个任务被分配到最早的阶段，此时所有所需能力都存在，且至少有一个是新的。评估了两种设置：部署（每个阶段使用新系统，隔离工具支架引发的遗忘）和自进化适应（基于记忆、提示词和代码的方法在各个阶段积累持久状态）。使用 ReAct (GPT-5)、Codex、Claude Code、AutoGen 和 DeLM 进行的实验表明，仅工具支架扩展就会降低之前已解决任务的性能（所有轴上的部署后向迁移均为负；在 ALE 上，智能体的性能下降最严重，为 -34.7%），适应增益在各个轴和环境之间不一致（MemToolAgent 在 EOG 工具上增加 8.4 个百分点，但在 ALE 上接近基线），且保留和适应可能存在冲突（GEPA 在 ALE 工具上表现出正后向迁移，同时伴随 -28.5% 的前向迁移）。主要局限性：启发式技能/智能体标注、缺乏显著性检验或声明的运行次数，以及低方差、高波动的智能体轴基线。
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Harness expansion alone degrades previously solved-task performance (harness-induced forgetting) even with fixed model weights and prompts, across tools, skills, and agents (e.g., -34.7% deployment BWT for agents on ALE, Fig. 6).
- Gains from self-evolving adaptation (memory, prompt, code methods) are inconsistent across harness stages, capability axes, and environments (MemToolAgent +8.4 points on EOG tools but near baseline on ALE, Table 2).
- Retention and adaptation are in tension: methods achieving positive backward transfer can show sharply negative forward transfer (GEPA: positive BWT but -28.5% FWT on ALE tools, Obs. 3).
- Failure mechanisms differ by axis: tools require experience that narrows search, skills require learning to engage the library (0% invocation for GPT-5-based systems, Fig. 5), and agents require delegation completeness rather than selection precision (precision ~90% flat while recall rises 79→89%, Fig. 7).
Strengths
Novel problem formalization with a clean, auditable construction
Section 3.1-3.2 formalizes harness evolution as nested cumulative harnesses H1⊆...⊆HT with two provable guarantees (feasibility, new-capability pressure), built by a deterministic frequency-ranked release pipeline that requires no LLM generation or manual verification — a genuine repositioning of non-stationarity from task stream to harness relative to Table 1's three benchmark families.
Factorial breadth across axes, environments, and method families
17 streams x 2 source environments (EOG/ALE) x 3 axes (Tables 2, 4, 6), evaluating 5 deployment systems (ReAct GPT-5, Codex, Claude Code, AutoGen, DeLM) and 7 adaptation methods spanning memory (ReasoningBank, MemToolAgent, G-Memory, LEGOMem), prompt (GEPA), and code (Meta-Harness) families.
Mechanism-level analyses beyond leaderboard reporting
Honest scoping and reporting discipline
Weaknesses
Skill and agent ground truth rests on unvalidated heuristics
Statistical reporting is thin
Fragile agent-axis baselines and censored evidence
Single canonical release ordering, unreconciled summary metrics
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
EvoHarnessBench (arXiv 2609.04280v2, Salesforce AI Research) targets a gap the authors argue is unaddressed: existing agent benchmarks either fix the harness (τ-bench, SkillsBench, MultiAgentBench; Table 1), study agent-driven capability growth (SkillLearnBench, SkillFlow), or vary the task stream (AgentCL, ContinualBench, CMRBench). Here the non-stationarity lives in the externally supplied harness — tools, skills, specialist agents. Central claims: (1) harness expansion alone causes 'harness-induced forgetting' of previously solved tasks; (2) gains from self-evolving adaptation are inconsistent across stages, axes, and environments; (3) retention and adaptation can trade off. Scale: 17 streams of 3-6 stages, 802 unique tasks (1,510 axis-specific examples), 520 tools, 42 skills, 62 agents, built deterministically from EnterpriseOps-Gym (EOG) and Agentic Last Exam (ALE).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is harness-induced forgetting?
Performance loss on tasks an agent previously solved, caused only by growth of the externally supplied harness — more tools, skills, or specialist agents become available — while model weights and prompts stay fixed. EvoHarnessBench isolates this with a deployment evaluation that instantiates a fresh system at each harness stage.
What is EvoHarnessBench built from?
It deterministically converts two verifier-based agent benchmarks — EnterpriseOps-Gym (stateful enterprise workflows) and Agentic Last Exam (diverse software-tool tasks) — into 17 cumulative harness streams of 3-6 stages, totalling 802 unique tasks instantiated as 1,510 axis-specific examples across 520 tools, 42 skills, and 62 specialist agents.
What are the paper's main findings?
Three gaps: (1) harness expansion alone degrades previously solved tasks (negative deployment backward transfer on all axes, worst -34.7% for agents on ALE); (2) gains from memory/prompt/code adaptation are inconsistent across axes and environments; (3) retention and adaptation can trade off — GEPA shows positive backward transfer on ALE tools alongside -28.5% forward transfer.
Is code or data available for EvoHarnessBench?
The paper links a project page (mas-orchestra.salesforceresearch.ai/evoharness/) and fully documents the deterministic construction with per-stage statistics in Appendix B, but an explicit code/data release statement does not appear in the sampled text — availability may be stated in sections skipped by sampling.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with three [...sampling gap...] breaks; content inside the gaps (parts of Sections 4.2-4.3, Appendix A, F.1, and I.2) was assessed only via surrounding context, so absence claims were avoided where gap content could exist.
- No explicit code/data-release statement appeared in the sampled text; codeAvailable=false reflects absence of evidence in the excerpt, not confirmed absence in the full paper.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.