HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei, Qingshui Gu, Yuxuan Zhang, Zexuan Wang, Chen He, et al.
cs.SE · cs.CL
Quick verdict
HarnessDev 发现 LLMs 能够构建可运行的智能体 harness，但在代码/搜索方面落后于人类系统，进化收益不稳定；严谨、坦诚的基准研究。
Summary
HarnessDev 是一个将智能体评估从任务输出转移到可运行基础设施的基准测试：创建者 LLM 必须从一个故意设定的弱、零分种子（RQ1，创建）构建一个完整的智能体 harness，然后利用下游执行反馈迭代地改进他们自己的 harness（RQ2，进化）。每个冻结的 harness 都在五个基准（SWE-Pro、Terminal-Bench 2.1、MLE-bench、EQ-Bench3、BrowseComp，共 2,207 个下游实例）上评估其能力（held-out 基准上的任务成功率）和效率（执行器 token 成本），且创建者与执行者模型是分离的。在自我评估下，来自 Opus 4.8 的 harness 领先（四个基准平均 67.8），而所有这些在代码和搜索/研究方面仍远逊于成熟的人工参考，在写作和 ML 实验方面则与参考持平或超越。执行成本差异高达 19 倍，但并无可靠的评分收益。产物分析揭示了普遍存在的缺陷：在 26,679 条轨迹中无检查点事件，2,325 个数据提交中有 441 个退化且未检测到，587 个写作特征中有 124 个为死代码。进化产生了反馈集增益（例如 Gemini +8.8），但在 630 个 held-out 任务上缩减（例如 +2.70），在固定执行器下可能会倒退（GPT-5.5 为 -10.32），并且在运行时模型之间的迁移性较差。局限性包括未配对的人工参考（外部的，未重新运行）、稀疏的 avg@3 重复以及繁重的 API 成本限制了独立验证。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLM-built harnesses from a zero-scoring seed match or exceed human references on writing and MLE-bench but remain substantially behind on code and search/research (Tables 3–4).
- Execution-token cost varies by up to ~19× across harnesses and does not reliably predict downstream score.
- Evolution improves the visible feedback pair in every lineage (+1.1 to +13.9) but held-out-630 changes span −10.32 to +4.44, with three of four fixed-executor lineages regressing.
- Harness quality is strongly executor-dependent; hard-coded budgets and stopping rules break portability (e.g., Opus SWE-Pro 69.3→33.0 under Gemini).
- Structural defects are pervasive: zero checkpoint events in 26,679 trajectories, 441/2,325 undetected degenerate Data submissions, and 124/587 dead Writing features.
Strengths
Genuinely novel evaluation axis with a clean formal setup
Shifts the unit of evaluation from task answers to runnable infrastructure (Eq. 1): the creator model builds H from a policy-free zero-scoring seed (Sec. 3.2), and the executor model is separated and frozen before evaluation — enabling real transfer analysis rather than conflated self-play.
Unusually honest, self-auditing reporting
External human references are asterisked as not re-run (Table 3); collapsed replicas get post-hoc clean sensitivity means (Table 4 ‡: Opus SWE-Pro 49.1; DeepSeek 43.8/57.3); a zero-harness outlier is disclosed (GPT-5.5 EQ-Bench3 46.5 vs 69.7 excluding it); 441/2,325 undetected degenerate Data submissions are reported against the authors' own systems.
Forensic artifact analysis beyond scores
Disciplined Evolution protocol separating visible from held-out gains
Scale and coverage
Weaknesses
Human references are unpaired external numbers
Thin replication with no uncertainty quantification on main rankings
Self-Eval confound only partially mitigated
RQ2 held-out generalization rests on one benchmark
Replication cost and unconfirmed code release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
HarnessDev (arXiv:2609.01437; ByteDance Seed with SUTD, Georgia Tech, M-A-P, TokenWave.AI) reframes agent evaluation: instead of scoring a model's answers under a fixed scaffold, it scores the scaffold the model builds. A creator LLM LC working in a development environment D must turn a weak seed into a runnable harness H; Eq. (1) then freezes H and hands it to a separate executor LLM LE. The seed (Sec. 3.2) is deliberately policy-free — no loop, tool policy, context management, state, verifier, or stopping rule — and scores 0.0 everywhere, so any nonzero score must come from creator-added logic. RQ1 (Creation) tests building from the seed plus 1–3 development cases; RQ2 (Evolution) tests improving one's own harness from downstream feedback. Harnesses are abstracted as H = ⟨E,T,C,S,L,V⟩ (execution, tools, context, state, lifecycle, verification). Evaluation is two-axis: capability (held-out success on SWE-Pro, Terminal-Bench 2.1, MLE-bench, EQ-Bench3, BrowseComp; 2,207 instances) and efficiency (executor tokens). Headline claims: model-built harnesses match or exceed human references on writing and ML experimentation but lag on code and search/research; cost varies ~19-fold without reliably buying score; evolution gains are unstable and executor-dependent.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the HarnessDev benchmark?
HarnessDev (arXiv:2609.01437) evaluates whether LLMs can act as agent-harness engineers: a creator model builds a complete execution harness from a weak zero-scoring seed (Creation) and then iteratively improves its own harness using downstream benchmark feedback (Evolution). Frozen harnesses are scored on capability and token cost across SWE-Pro, Terminal-Bench 2.1, MLE-bench, EQ-Bench3, and BrowseComp — 2,207 downstream instances in total, with the creator and executor models deliberately separated.
Can LLMs build their own agent harnesses as well as humans?
Not yet overall. Under self-evaluation, harnesses built by Opus 4.8 average 67.8 across four benchmarks versus an 86.2 human-system reference. Model-built harnesses match or exceed references on writing (84.6 vs 83.7 on EQ-Bench3) and ML experimentation (32.9 vs 24.0 MLE-bench medal rate), but lag clearly on code (69.3 vs 80.0 SWE-Pro) and search/research (52.4 vs 92.2 BrowseComp).
Do self-improving harness gains generalize to new tasks?
Only partially. Every lineage improved its visible feedback pair (gains from +1.1 to +13.9 points), but changes on 630 held-out SWE-Pro tasks ranged from −10.32 to +4.44, and under a fixed Gemini executor three of four lineages regressed — GPT-5.5 by −10.32 points — showing gains often specialize to the feedback set or executor.
Which models built the best harnesses?
Claude Opus 4.8 created the strongest harnesses under self-evaluation (67.8 four-benchmark average), ahead of Gemini 3.1 Pro (55.6) and GPT-5.5 (55.1); Seed 2.0 Pro was weakest (22.8). Rankings shift substantially when the executor model is swapped (e.g., Opus SWE-Pro falls from 69.3 to 33.0 under Gemini), revealing strong harness-executor coupling.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with three [...sampling gap...] regions (mid-paper sections and parts of the appendices); judgments rely on the visible protocol, tables, and case studies, and claims about gapped regions are avoided.
- Full code/data release could not be confirmed from the excerpt; only audit-artifact release and a project page URL are stated explicitly, so codeAvailable=false reflects what the provided text states, not a verified absence.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.