An Empirical Study of Harness Design for Coding Agents
Run-Ze Fan, Zihao Zhang, Simin Ma, Yebowen Hu, Shouju Wang, Kaiqiang Song, Fei Liu, Hamed Zamani, et al.
cs.AI · cs.CL · cs.LG · cs.SE
Quick verdict
Controlled 176-setting ablation: harness-component effects hinge on model, budget, task type. Credible; single runs and bundled action space limit reach.
Summary
This paper dissects coding-agent harnesses into three independently ablatable components—planning, action space, and context management—on top of a fixed ReAct execution loop, running 176 matched settings across four models (Nemotron-3 30B/120B/550B and Mistral-Medium-3.5-128B) on SWE-Bench Verified and Terminal-Bench 2.1. Context management spans five tiers (none; elision; elision+recall; summarization; staged T4) under four window budgets (32k–128k); planning and action space are ablated only at T4/128k. Success differences are tested with task-paired exact McNemar tests under Benjamini–Hochberg FDR control. The managed-versus-unmanaged gap shrinks from 35.7 to 2.7 percentage points as the window grows from 32k to 128k on SWE-Bench, tracking the collapse of T0 overflow failures (78.7%→8.7%); T4 achieves comparable accuracy at the lowest cost in seven of eight panels; the recall mechanism is rarely invoked (median zero calls) and does not beat elision alone. Planning boosts the weakest model (+11.6 pp on SWE-Bench for 30B) but mainly cuts cost (~30%) for stronger ones; predefined tools scaffold bash-weak models (+15.0 pp for 30B on SWE-Bench), while bash-only favors bash-capable models (550B: +3.6 pp at −53% cost). Limitations: one run per task, 89 Terminal-Bench tasks, ablations restricted to T4/128k, and an action-space change bundled with state tracking and diagnostics.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Context management's value grows as the context-window budget tightens, with most of its benefit coming from preventing overflow failures (managed−T0 gap: 35.7→2.7 pp on SWE-Bench from 32k→128k, tracking T0 overflow 78.7%→8.7%).
- Staged elision-before-summarization (T4) achieves the best accuracy–cost profile among the five context-management strategies (lowest cost in 7 of 8 model–benchmark panels), and lossless recall (M2) is rarely invoked (median zero calls/task) with no accuracy gain over elision alone.
- Planning acts as an accuracy scaffold for the weakest model (Nemotron-3 30B: +11.6 pp on SWE-Bench Verified, +4.5 pp on Terminal-Bench) but mainly as a cost saver for stronger models (~30–32% cost reduction with −0.4 to −2.0 pp success).
- Predefined tools scaffold bash-weak models (30B: +15.0 pp on SWE-Bench), whereas bash-only interfaces favor bash-capable models (Nemotron-3 550B: +3.6 pp at −53% cost on SWE-Bench; Mistral: +6.7 pp on Terminal-Bench but −23.2 pp on SWE-Bench), with the crossover set by task shell-centricity.
Strengths
Genuinely controlled component-level design
The execution loop, safety gates, post-edit diagnostics, and stuck detection are held fixed (§2.4) while one component varies at a time; 176 matched settings across 4 models × 2 benchmarks, and every comparison is against a matched T4/128k baseline rather than a foreign harness.
Exceptional reporting transparency
All system prompts (Figs 10–17), tool descriptions (Figs 21–31), the context-management algorithm (Algorithm 1), stub/summary fragments (Fig 19), and LLM-judge prompts (Figs 40–42) are reproduced verbatim in appendices — rare in this literature and directly enabling reimplementation.
Above-average statistical protocol
Mechanistic, human-validated trajectory analysis
Conditional, scope-limited conclusions
Weaknesses
Single run per task; stochasticity unquantified
Action-space intervention is a bundle
Capability axis conflates scale with family/training
Threshold confound in the T4-vs-others comparison
Ablation coverage restricted to T4/128k
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: An Empirical Study of Harness Design for Coding Agents
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a coding-agent harness?
The software layer around an LLM that turns model outputs into actions on a codebase: the execution loop, the tool interface, and the context-management policy. This paper shows each component's effect is conditional — on model capability, context-window budget, and task type — rather than universally beneficial.
Does context management (summarization/elision) actually improve coding-agent accuracy?
Mainly under tight context windows. In the paper, the success-rate gap between managed and unmanaged runs shrinks from 35.7 to 2.7 percentage points on SWE-Bench Verified as the window grows from 32k to 128k tokens, and most of the benefit comes from preventing runs from dying of context overflow.
Should coding agents use predefined tools or plain bash?
It depends on the model. Models weak at bash gain a lot from predefined tools (Nemotron-3 30B: +15.0 points on SWE-Bench Verified), while bash-capable models do better and cheaper with bash-only (Nemotron-3 550B: +3.6 points at −53% cost on SWE-Bench). Mistral preferred bash on Terminal-Bench but predefined tools on SWE-Bench.
Is planning (todo lists) useful for coding agents?
For weak models, yes: a persistent plan keeps trajectories alive long enough to attempt an edit (Nemotron-3 30B: +11.6 points on SWE-Bench Verified). For strong models it mainly removes redundant post-edit verification, cutting cost ~30% with little accuracy change.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are only partially recoverable from the extracted text (axis ticks and labels are garbled); figure-based claims were cross-checked against Tables 3–5 and 10–13 where possible.
- Portions of §4 (trajectory analysis narrative), Tables 6/8/9, and parts of Appendix C were truncated in extraction; some in-text analysis details may be missed.
- The affiliation line for affiliation '2' is garbled in extraction (Zoom per email domain), and Xiaoyang Wang appears in the body header and emails but not in the metadata author list.
- No code or artifact release statement appears in the extracted text; one may exist in unextracted sections or on the paper's project page.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.