Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails
Zhou Yu, Bin Bi, Shiva Kumar Pentyala, Shubham Mehrotra, Sougata Chaudhuri, Shilpa Bhagavath, Zeyuan Chen, Ran Xu, et al.
cs.AI
Quick verdict
Claims expert imitation breaks weak-model fit to an evolved harness; diagnosis credible but the +1.7pt fix rests on 3 runs, one student, confounded data.
Summary
The paper studies how to combine two adaptation levers for LLM agents: automated harness evolution (GEPA-style search over prompts, tools, hooks, and context management) and lightweight model fine-tuning (LoRA-SFT). On seven enterprise agentic benchmarks from Yang et al. (2026), evolving a harness around Qwen3-Coder-30B raises mean test success from 29.2% to 78.0%, and a Gemini 3.1 Pro expert reaches 93.6% under that same harness, suggesting expert imitation as the natural next step. Surprisingly, LoRA fine-tuning Qwen on the expert's successful full trajectories under the evolved harness regresses all seven tasks (mean −14.9 points, −4.2 to −29.9), while the identical recipe improves the baseline-harness setting (+6.3); a Gemma replication on one task shows the same regression. Failure-ontology analysis attributes the damage to planning-style drift (planning failures rise from 1.1% to 14.6% of failures) rather than lost knowledge or reduced scaffold use (domain-recipe adoption actually rises from 30.8% to 76.1%). The authors propose on-policy expert correction: an automated MLE-agent pipeline localizes the failing turn in the student's own rollout and has the expert rewrite only that turn (~400 correction rows plus ~50 self-pass rows). This yields +1.7 mean points over the evolved-harness baseline (79.7%), keeping planning failures near 1.8%. Limitations: three runs per cell without significance testing, one expert model, correction evaluated only on Qwen, and data-scale confounds between the imitation and correction arms.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A harness evolved for a weaker model transfers upward: the Gemini 3.1 Pro expert improves from 84.4% to 93.6% mean test success under it, while the Qwen student reaches 78.0% from 29.2% (Table 1 rows 1–4).
- LoRA fine-tuning the weaker student on the expert's full successful trajectories under the evolved harness regresses all seven tasks (mean −14.9 points, −4.2 to −29.9), whereas the identical recipe helps under the baseline harness (+6.3).
- The regression stems from planning-style drift, not knowledge loss: scaffold usage rises (domain recipe 30.8%→76.1%) while planning failures jump from 1.1% to 14.6% of failures (Tables 2–3).
- On-policy expert correction — rewriting only the failing turn in the student's own rollouts (~400 rows) — preserves model–harness fit and adds +1.7 mean points (78.0%→79.7%) with planning failures held at ~1.8%.
Strengths
Clean crossover design isolates the interaction effect
The identical imitation recipe helps under the baseline harness (29.2%→35.5%, +6.3, Table 1 rows 1/6) yet hurts under the evolved harness (78.0%→63.1%, −14.9, rows 2/5), pinning the failure on the imitation–harness-evolution interaction rather than imitation per se (§3.3.1).
Mechanistic diagnosis backed by two independent evidence streams
The adoption scan (Table 2: domain-computation recipe usage rises 30.8%→76.1% after imitation) and the failure-ontology reclassification (Table 3: planning failures jump 1.1%→14.6% of failures while the knowledge bucket shrinks) jointly support the 'knowledge transferred, planning fit broken' dissociation; Appendix D case studies (e.g., the 78-step payroll verify loop) make the mechanism concrete.
Cross-family replication of the negative result
Unusually complete training recipe
Honest scoping of the positive result
Weaknesses
Statistical fragility of the headline fix
Imitation vs correction comparison is confounded
Asymmetric generality: failure shown on two students, fix on one
Reporting inconsistencies and post-hoc choices
LLM-as-judge failure labels unvalidated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This Salesforce AI preprint (arXiv 2609.09134, Sep 2026) studies the interface between two adaptation levers for domain-specific LLM agents: automated harness evolution and lightweight weight tuning. It makes three claims. (1) Evolved harnesses transfer upward — under a harness evolved for Qwen3-Coder-30B, that model rises 29.2%→78.0% mean test success while Gemini 3.1 Pro reaches 93.6% from 84.4% (Table 1, rows 1–4). (2) LoRA imitation of the expert's full successful trajectories under the evolved harness backfires on all seven tasks (mean −14.9, range −4.2 to −29.9), while the identical recipe helps under the baseline harness (+6.3) — isolating an imitation×harness-evolution interaction. (3) On-policy expert correction, in which an automated MLE agent localizes the failing turn of the student's own rollout and the expert rewrites only that turn, adds +1.7 mean points (79.7%) while keeping planning failures near baseline (1.1%→1.8% of failures).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why does fine-tuning on a stronger model's trajectories hurt performance?
In this paper's setting the harm appears only when the harness was already evolved around the weaker model's own planning style: the student copies the expert's planning strategy without the competence to execute it, so planning failures jump from 1.1% to 14.6% of failures and it no longer matches the tuned harness. Under an unevolved baseline harness, the same fine-tuning actually helps (+6.3 points), showing the problem is the interaction with harness evolution, not imitation itself.
What is on-policy expert correction?
It is a training-data pipeline in which an automated MLE agent finds the single turn where the student model's own rollout goes wrong, and a stronger expert model rewrites only that turn while leaving all other steps untouched. The student is then LoRA fine-tuned on these minimally edited trajectories (~400 correction rows plus ~50 self-pass rows), preserving its planning style. It gained +1.7 points on average over the evolved-harness baseline in the paper.
How much does harness evolution help weaker models?
On seven enterprise agent tasks, evolving the harness lifted Qwen3-Coder-30B from 29.2% to 78.0% mean test success (+48.8 points), and a Gemini 3.1 Pro expert reached 93.6% under the same harness, up from 84.4% under the default one.
Is code or data available for this paper?
The paper does not state a code or data release. It builds on the public task suite and harness-optimization framework of Yang et al. (2026) and uses public Qwen and Gemma checkpoints plus the Gemini API, but the correction pipeline and fine-tuning code are not described as released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's diagram content cannot be verified from text extraction (its caption is present, the visual is not).
- The provided metadata's author list is shorter than the full text's, which additionally lists Phil Mui, James Zhu, and Sitaram Asur.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.