Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite
cs.AI · cs.CL · cs.SE
Quick verdict
Claims no average harness advantage; candid, well-instrumented study, but underpowered with a post-hoc headline finding and unresolved billed costs.
Summary
This single-author empirical study asks whether a vendor-native agent harness outperforms a neutral harness on the same model. Using a private, contamination-controlled suite of 256 tasks (179 repository tasks mined from four internal codebases plus 77 post-cutoff contest problems), the author ran paired contrasts: claude-opus-4-8 under claude-agent-sdk versus deepagents, and gpt-5.5 under openai-codex versus deepagents, with Gemini-3.5-flash and DeepSeek-V3.2 side cells; 792 of 800 planned runs were graded by an isolated Docker oracle. Neither contrast resolves an average advantage (Opus: -1.25 pp, bootstrap CI [-10.0, +7.5]; GPT-5.5: +1.25 pp, CI [-4.4, +6.9]). A post-hoc workload split on Opus shows opposite signs (repository -9.0 pp, contest +23.7 pp, permutation p=0.003), but the partition was chosen after seeing the data. A usage-semantics defect that double-counted cache tokens invalidated the original cost claims; re-pricing from raw per-turn usage yields 1.3-1.6x cost per solved task for the neutral harness on Opus and 1.2x on GPT-5.5, with the Anthropic billed ordering unresolved (58 runs lack usage records). Key limitations: confidence intervals spanning 11-17 pp exceed practical effect sizes, the task pool oversamples screening-discordant tasks, pre-registration is weaker than first claimed, and the tasks remain private.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On the same model, neither contrast shows a resolvable average solve-rate advantage for the vendor-native harness (Opus 4.8: -1.25 pp, CI [-10.0, +7.5]; GPT-5.5: +1.25 pp, CI [-4.4, +6.9]) on the 80-task private pool.
- On Opus 4.8 the harness effect reverses sign between repository (-9.0 pp, 61 tasks) and contest (+23.7 pp, 19 tasks) strata with permutation p=0.003, but the partition is post-hoc and requires a designed replication.
- Correct output and autonomous completion are distinct endpoints: 22 of 81 runs cancelled at the 1,200 s ceiling had produced passing patches, and the neutral harness hit the ceiling far more often.
- Re-priced from raw per-turn usage at frozen list prices, the neutral harness cost 1.3-1.6x more per solved task on Opus 4.8 and 1.2x on GPT-5.5, with the Anthropic billed ordering unresolved.
- A telemetry normalizer that assumes one SDK's input-token convention can double-count cache tokens on another, inflating recorded deepagents-cell costs 4.8-7.7x - a reusable methods lesson for agent cost reporting.
Strengths
Paired same-model design isolates the harness variable
Contrasts C1-C2 (claude-agent-sdk vs deepagents on claude-opus-4-8) and C3-C4 (openai-codex vs deepagents on gpt-5.5) share one substrate: one KVM microVM per run, identical seeds/prompts, append-only event ledger, and a runtime drift gate that verifies the served model per leg (§2.2, §3.1), including a live catch of OpenAI serving gpt-5.5-2026-04-23 under a pinned alias.
Exceptional self-audit transparency
§5.1 documents the study's own cache-token double-count with a source-to-ledger reconciliation (Table 7: exact for 693/703 rows), Appendix F retracts each August claim item-by-item, and §4.3/Appendix C disclose that the multiplicity family was changed after review, moving the interaction q from 0.10 to 0.013.
Cost claims reconciled against real money
Endpoint separation is a substantive finding
Statistical reporting discipline
Weaknesses
Underpowered for deployment-relevant effects
The headline interaction is post-hoc on a small stratum
Cost conclusion is asymmetric in strength across vendors
Selection and generalizability limits
Pre-registration is weaker than originally claimed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This single-author study (arXiv:2609.11987, revised 8 Sep 2026) asks whether a vendor-native agent harness beats a neutral harness on the same model. Cells C1-C2 pair claude-agent-sdk against deepagents (LangGraph) on claude-opus-4-8; C3-C4 pair the openai-codex SDK against deepagents on gpt-5.5; C5-C6 are Gemini-3.5-flash and DeepSeek-V3.2 side cells. The testbed is a private, contamination-controlled suite: 179 SWE-bench-pattern repository tasks mined from four internal codebases (AccessCtl 110, FieldSvc 40, IdentityApp 27, BookingSvc 2) plus 77 LeetCode/AtCoder problems post-dating a mechanically derived eligibility date (2026-03-02 = newest binding cutoff 2026-01-31 + 30 days). Central claims: (1) no resolvable average harness advantage in either contrast; (2) a post-hoc workload interaction on Opus; (3) correctness and autonomous completion are distinct endpoints; (4) neutral-harness cost premiums of 1.3-1.6x (Opus) and 1.2x (GPT-5.5) per solved task; (5) a cache-token double-count defect as a methods result. This revision formally withdraws the August 2026 cost conclusions (Appendix F).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did the study find that vendor harnesses beat neutral harnesses like deepagents?
No. Holding the model fixed, the solve-rate difference was -1.25 percentage points for Claude Opus 4.8 (native vs deepagents) and +1.25 points for GPT-5.5, both with 95% confidence intervals including zero. On this 80-task private pool, neither harness shows a resolvable average advantage; the interesting differences appear only in an exploratory workload split and in completion behaviour and cost.
What was the telemetry defect in this paper?
The study's cost pipeline assumed every SDK reports input tokens exclusive of cache tokens. Anthropic does, but the LangChain metadata surfaced by deepagents reports them inclusive, so the normalizer added cache tokens a second time, overstating recorded costs for the deepagents cells by roughly 4.8-7.7x. The September 2026 revision re-derives all costs from raw per-turn usage events and withdraws the original 2.6x/1.8x premium claims.
How much more expensive is deepagents per solved task?
On observed usage at frozen list prices, deepagents cost about 1.3-1.6 times as much per solved task as claude-agent-sdk on Opus 4.8, and about 1.2 times as much as the codex SDK on GPT-5.5. Caveat: on the Anthropic account, 25-29% of billed spend had no usage record, so the true billed ordering of the two Opus cells cannot be confirmed from this study.
Is the paper's benchmark dataset public?
No. The 256 tasks (179 repository tasks from private codebases and 77 post-cutoff contest problems) stay private to preserve contamination control. The orchestrator, grading oracle, reanalysis code and derived aggregates are released, and the authors offer the suite to artifact-evaluation committees under NDA.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: several middle passages are marked as skipped ([...sampling gap...]), including parts of §2.3, §4.3, §5.4, §6 and Appendix G. All main tables (1-18) and appendices A-H are visible, so the core quantitative claims were verifiable.
- The metadata abstract is cut mid-sentence ('On the Anthropic accou...'), but the complete abstract appears at the head of the extracted paper text, so no conclusion rests on the truncated copy.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.