Harness-Zero: Harness Distillation via Agent-as-Harness
Haoran Ye, Yuxing Lu, Haonan Dong, Zhaochen Su, Guojie Song
cs.AI · cs.CL · cs.NE
Quick verdict
Claims agent-review distillation lifts Qwen3.5-9B from 23.3% to 44.3% under a minimal harness, beating the evolved harness; single-run evidence.
Summary
The paper studies harness distillation: transferring the behavioral gains of a domain-optimized agent harness (tools, middleware, skills, memory) into model weights, so deployment needs only a minimal fixed harness (mini-SWE-agent with one bash tool). Because source and target harnesses differ in action space, direct imitation fails; Harness-Zero instead wraps the student with a 'harnessing agent' that reviews each proposed response and either passes it or substitutes a minimal correction expressed in the target harness's action space, guided by a reference harness adapted from the evolved one. SFT on these reviewed trajectories (with reviewer-perspective reasoning masked) internalizes the behaviors. Across SpreadsheetBench, AppWorld, and USPTO, agent-as-harness with an evolved reference harness averages 81.1% on frontier models versus 78.1% for code-as-harness (Table 1). Distilling into Qwen3.5-9B raises macro-average success from 23.3% to 44.3%, exceeding the 41.7% achieved by the base model with the evolved harness still attached (Table 2). Ablations on USPTO (Table 3) show teacher trajectories (12.0%) or oracle answers (15.0%) distill poorly compared to harness-guided review (30.0%). The distilled model recovers 82.3% of 28 harness-exclusive behaviors (Table 4). Limitations: single-run evaluations on small test sets without variance estimates, per-domain specialists rather than a generalist, dependence on a capable harnessing model, 2.4x collection latency, and incomplete internalization of domain knowledge (USPTO trails the evolved harness).
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Agent-as-harness outperforms code-as-harness for frontier models using the same evolved harness, 81.1% vs 78.1% averaged over six benchmark-model settings (Table 1).
- SFT on harnessing-agent-corrected trajectories lifts Qwen3.5-9B's macro-average success from 23.3% to 44.3% under the minimal harness alone, exceeding the 41.7% obtained with the evolved harness attached (Table 2).
- Harness-guided review is the effective supervision signal: teacher trajectories, trajectories generated under h*, empty-K review, and oracle-answer review all distill to 3–15% versus 30.0% for the full method on USPTO (Table 3).
- The distilled model recovers an average 82.3% of 28 harness-exclusive behavioral patterns across the three domains (Table 4).
- The approach requires a sufficiently capable harnessing model; its advantage over code-as-harness reverses on weaker models (Appendix F).
Strengths
Novel, well-motivated problem formulation
Harness distillation addresses real harness-lock-in at deployment; the headline claim is directly tested in Table 2, where the distilled model under h alone (44.3% macro-average) exceeds the base model with the evolved harness h* still attached (41.7%).
Exceptionally well-controlled ablation suite
Table 3 isolates the supervision source with six controls: teacher rollouts under h (52.0% collection → 12.0% test), teacher under h* (62.0% → 3.0%), student under h* (39.4% → 12.0%), empty-K review (44.2% → 11.0%), oracle-answer review (98.6% → 15.0%) vs the full method (59.4% → 30.0%), cleanly decoupling collection success from distillation value.
Mechanistic behavioral evidence beyond aggregate scores
Above-average reproducibility disclosure
Honest capability and cost analysis
Weaknesses
No variance reporting anywhere
Per-domain specialists, not the generalist the abstract promises
Missing rejection-sampling baseline
Contamination and teacher-knowledge confounds unaddressed
Aggregation of the 82.3% recovery figure is opaque
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Harness-Zero (arXiv 2609.24974) formulates **agent harness distillation**: behaviors induced by a domain-optimized harness h* (tools, middleware, skills, memory, evolved per §3.1) are transferred into model weights so deployment needs only a fixed minimal harness h (mini-SWE-agent, one bash tool). The enabling mechanism is **agent-as-harness** (§3.2): a harnessing agent intercepts each student response, PASSes or REPLACEs it with a minimal correction expressed in h's action space, guided by a private reference harness K adapted from h*. SFT on the reviewed trajectories (reviewer-voice reasoning masked, Appendix D) internalizes the behaviors. Three claims are tested on SpreadsheetBench Verified, AppWorld, and USPTO: (1) agent-as-harness beats code-as-harness at inference (81.1% vs 78.1% average over six settings, Table 1); (2) distillation lifts Qwen3.5-9B from 23.3% to 44.3% macro-average under h alone, exceeding 41.7% with h* attached (Table 2); (3) 82.3% average recovery of 28 harness-exclusive behaviors (Table 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Harness-Zero in one sentence?
A fine-tuning method that transfers the behaviors of a domain-optimized agent harness (tools, middleware, skills, memory) into model weights by having a second 'harnessing agent' correct a student model's responses during trajectory collection, so that deployment needs only a minimal harness with a single bash tool.
How much does Harness-Zero improve performance?
On Qwen3.5-9B it raises macro-average task success across SpreadsheetBench, AppWorld, and USPTO from 23.3% to 44.3% under the minimal harness — above the 41.7% the base model reaches even with the optimized harness attached. These are single-run results on held-out test splits of 100 tasks or 56 scenarios.
What is the difference between agent-as-harness and code-as-harness?
Code-as-harness runs the model directly under an evolved code harness that shapes its action space. Agent-as-harness instead runs it under a fixed minimal harness while a second agent instance reviews each proposed action and, when needed, replaces it with a corrected version in the same action space, guided by an adapted reference harness.
Is the code and data available for Harness-Zero?
The paper lists a public GitHub repository (github.com/metaevo-ai/harness-zero) and uses public benchmarks (SpreadsheetBench Verified, AppWorld, USPTO). Reproduction additionally requires API access to proprietary models (GPT-5.6 Sol, DeepSeek-V4-Pro, Kimi K3, Qwen3.5-9B) and GPU fine-tuning (A100, LoRA rank 32).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 and appendix code listings are available only as caption/inline text; visual layout and figures could not be assessed.
- The GitHub repository's actual contents were not verifiable from the text; codeAvailable reflects the paper's claim.
- The abstract in the supplied metadata is truncated mid-sentence, but the complete abstract appears in the body text.
- Appendix F reports exact USPTO-only comparisons; capability trends on the other two domains are not given, limiting generalization of that analysis.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.