Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement
Haoyang Yan, Min-le Su, Hangfan Zhang, Zhanhao Li, Chen Zhang, Shao Zhang, Yang Chen, Lei Bai, et al.
cs.AI
Quick verdict
Wraps coding agents in planner-coder-QA loops for multi-day dev; consistent gains on 3 benchmarks, but single-run, judge-scored results need replication.
Summary
HoH is an orchestration layer that wraps unmodified coding-agent harnesses (Codex CLI, OpenCode, Pi) in a repeating planning–coding–testing loop with three roles (Project Planner, Developer, QA Tester) and two persistent state channels: the evolving artifact and a structured evidence bundle. A deterministic runtime freezes candidates, enforces per-role permissions, and normalizes QA reports into claim–evidence records that steer the next plan. On 45 GameCraft-Bench tasks, 15 FrontierSWE tasks, and ProgramBench, HoH@3 beats the vanilla harness under all three harness–model pairs: GameCraft overall rises 49.58→71.52 (Codex+GPT-5.5), 26.90→48.98 (OpenCode+DeepSeek-V4-Pro), and 42.16→58.78 (Pi+MiniMax-M3); Codex's FrontierSWE dominance reaches 71%, and 72.67% at HoH@10. In a pass-matched comparison, HoH@2 (64.84, 5.67M tokens) exceeds three-pass Vanilla Continuation (58.24, 6.33M tokens), and ablations attribute the gains to plan updating, evidence feedback, and warm-starting. A multi-day, 70-plus-loop case study autonomously develops a playable FPS game with human involvement limited to restoring connectivity. Limitations: one run per task–condition with no significance testing (bootstrap CIs appear in only one analysis), scorer/judge independence not detailed in the provided text, reliance on proprietary models at substantial cost, and abstract relative-gain figures whose derivation is not shown.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 4
● low 0
5 strengths · 0 weaknesses · 0 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- HoH@3 consistently outperforms the standalone harness across three benchmarks and three harness–model pairs, with absolute GameCraft-Bench gains of 16.62–22.08 points.
- Gains are not explained by extra compute: at matched development passes HoH@2 (64.84, 5.67M tokens) exceeds three-pass Vanilla Continuation (58.24, 6.33M tokens) on GameCraft-Bench.
- All three mechanisms matter: removing plan update, evidence feedback, or warm-starting drops the GameCraft-Bench mean from 71.52 to 63.39–65.23 (Table 22).
- HoH sustains multi-day autonomous development: 70+ loops produced a human-playable FPS game with 65 of 81 issues closed and no human role beyond restoring network/API availability.
- For Codex+GPT-5.5 on FrontierSWE, dominance keeps rising with iterations, reaching 72.67% at HoH@10 against a Vanilla baseline of 27.33%.
Strengths
Compute-controlled baseline directly answers the 'just run longer' objection
Table 2 and Fig 9 pit HoH against Vanilla Continuation at 1–3 matched development passes with per-task token accounting (Table 21): HoH@2 reaches 64.84 with 5.67M tokens versus 58.24 at 6.33M for three-pass continuation, quantified as 3.77 vs 2.32 points per extra million tokens (Eq. 12)
Consistent gains across three independent harness–model pairs (Tables 1, 14–16)
GameCraft-Bench Overall rises 49.58→71.52 (Codex+GPT-5.5), 26.90→48.98 (OpenCode+DeepSeek-V4-Pro), 42.16→58.78 (Pi+MiniMax-M3); HoH@3 wins every reported task category, and OpenCode with HoH@3 even exceeds Codex Vanilla in some categories
Evaluation isolation and anti-gaming design
Unusually transparent result reporting
Methodologically careful player-experience protocol
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
HoH (Shanghai AI Laboratory; arXiv 2609.01481) targets autonomous software development: given only a high-level specification, produce a complete, usable system with no human in the loop. The contribution is an orchestration layer that wraps unmodified harnesses (Codex CLI 0.142.5 + GPT-5.5 high, OpenCode 1.14.30 + DeepSeek-V4-Pro, Pi 0.80.10 + MiniMax-M3) in planning–coding–testing loops. Authority is split across three roles: a Project Planner (read-only) turns the spec plus a structured evidence bundle into a bounded development document; a Developer warm-starts from the prior artifact as sole writer; a QA Tester evaluates a frozen read-only candidate with complementary black-/white-box tests, returning claim–evidence records. A deterministic runtime enforces permissions and output schemas. Four headline claims: HoH@3 beats Vanilla everywhere tested (Table 1); the gain is not a compute artifact (pass-controlled Table 2); all three mechanisms — plan update, evidence feedback, warm-start — are needed (Table 22); and 70+ loops autonomously yield a playable FPS game (§5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 0 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Harness-of-Harness (HoH)?
HoH is an orchestration framework that wraps unmodified coding-agent harnesses (Codex CLI, OpenCode, Pi) in an outer loop of planning, coding, and independent testing. Three roles (Project Planner, Developer, QA Tester) collaborate, and two state channels — the evolving software artifact and a structured evidence bundle — persist across iterations, letting agents autonomously improve software over dozens of loops without human intervention.
How much does HoH improve over a plain coding agent?
After three iterations, HoH@3 beat the vanilla harness on all three benchmarks and all three harness–model pairs tested. GameCraft-Bench overall rose from 49.58 to 71.52 (Codex+GPT-5.5), 26.90 to 48.98 (OpenCode+DeepSeek-V4-Pro), and 42.16 to 58.78 (Pi+MiniMax-M3); Codex's FrontierSWE dominance rose from 44% to 71%; ProgramBench pass rates rose by 6.09–16.85 points.
Does HoH win just because it uses more compute?
The authors ran a pass-controlled comparison (Table 2): at one, two, and three development passes, HoH scored 59.71/64.84/71.52 versus Vanilla Continuation's 49.58/54.99/58.24 on GameCraft-Bench. HoH@2 beat three-pass continuation (64.84 vs 58.24) while using fewer tokens (5.67M vs 6.33M). However, the control covers one configuration and continues the same session.
What was built in the multi-day autonomous demo?
Given only high-level product requirements, HoH ran over 70 iterations across multiple days to build Fusepoint, a first-person shooter with a storyline, combat systems, menus, audio, and a human-playable build, using Codex CLI with GPT-5.6-Sol and Godot. Human involvement was limited to restoring network/API availability; 65 of 81 tracked issues were closed by loop 70.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with [...sampling gap...] regions (parts of §3.3–3.4, qualitative-analysis continuation, PXI results table, Appendix A/B portions); this review avoids asserting absence for any skipped content.
- Who or what assigns GameCraft-Bench rubric scores and FrontierSWE official rewards is not specified in the provided text; it may appear in skipped sections or in the cited benchmark papers, so the judge-independence flag is phrased as unresolved rather than missing.
- The abstract's relative-gain figures (52.25% average, 82.86% maximum) are not derived anywhere in the provided text; the average is numerically consistent with pooling relative gains across the nine benchmark–configuration cells, but this cannot be confirmed from the excerpt.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.