What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents
Chenqian Le, Jiayi Cheng, Qijia He, Runhao Li, Yinghao Li, Xupeng Chen
cs.AI
Quick verdict
Claims cross-harness GRPO credit adds no portable gain over within-harness grouping in coding-agent RL; rigorously controlled null with honest power limits.
Summary
The paper isolates one design decision in multi-harness agent RL: where the GRPO relative-advantage boundary sits. Cross pools the rewards of four production coding harnesses (Aider, OpenHands, Qwen Code, SWE-agent) into one advantage group per task; Within normalizes within each task-harness pair. Arms branching from a single Qwen3-8B warm start replay an identical frozen manifest of 81,216 records (183 SWE-Gym tasks) with the same 81,200 optimizer steps, and every checkpoint is graded by a sealed SWE-bench Verified oracle on the four source harnesses (24,000 episodes, 99.1% verdict coverage) plus a minimal weak-ReAct harness held out of training. The evaluation harness moves mean solve rate from 2.14% to 9.27% (factor 4.3), dwarfing the 1.16x spread across training recipes. On the held-out harness, Cross minus Within is +0.25 pp [-0.48, +1.02] at eight attempts and +0.16 [-0.41, +0.72] pooled over three seeds whose individual estimates (+0.62, +0.05, -0.20) change sign. An out-of-fold classifier recovers the generating harness from Cross's advantage (+4.48 pp above a shuffled-label null) but not from Within's (+0.02), yet this contamination leaves no detectable mark on held-out scores or action distributions; both rules concentrate gains on source harnesses. Limitations: one model family, a 183-task training pool, 1-10% solve rates, and a minimum detectable difference near 1 pp, so smaller true effects cannot be excluded.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across 24,000 sealed SWE-bench Verified evaluations, the evaluation harness moves mean solve rate from 2.14% to 9.27% (factor 4.3), while the training recipe spans only a factor of 1.16.
- With exposure and budget held fixed, Cross-harness GRPO grouping yields no detectable held-out gain over Within grouping: +0.25 pp [-0.48, +1.02] at eight attempts and +0.16 [-0.41, +0.72] pooled over three seeds whose per-seed estimates change sign.
- An out-of-fold classifier recovers the generating harness from Cross's advantage at +4.48 pp above its shuffled-label null while Within's advantage carries none (+0.02 pp), yet this harness identity leaves no detectable mark on held-out scores or action distributions.
- Both grouping rules concentrate their gains on the source harnesses (largest on OpenHands), a pattern of configuration adaptation rather than portable capability, and re-collecting half the training data on-policy does not change it.
Strengths
Clean isolation of a single design variable
Arms replay the identical frozen 81,216-record manifest with same tokens, rewards, loss masks and 81,200 optimizer steps (Appendix E, Table 7, ESS >= 0.993), so the GRPO grouping boundary is the only difference - unlike the systems it critiques (Appendix K: OpenForgeRL varies SFT and RL harnesses jointly; Polar evaluates checkpoints only on their own training harness).
Preregistered, cluster-robust statistical protocol
Six held-out contrasts fixed before checkpoints existed; task-level paired bootstrap CIs (10,000 draws); Bonferroni 99% CI for the seen contrast; explicit MDE80 at every budget (Appendix L, Table 3); >=95% verdict-coverage gate with abort-on-fail; infra failures quarantined via four-state verdicts.
Novel and mechanistically informative diagnostic
Honest null-result framing
Exemplary infrastructure forensics
Weaknesses
Power ceiling on the headline null
Residualized control failed as implemented
Narrow, low-competence training regime
Single-checkpoint evidence for secondary arms
Action-labeller noise near the smallest effect
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper isolates one design decision in multi-harness agent RL: where the GRPO relative-advantage boundary sits. Within normalizes rewards per task-harness pair; Cross pools all four harnesses per task so their outcomes compete for credit. From a single Qwen3-8B supervised warm start (8,841 records, 186 tasks), arms replay a frozen 81,216-record manifest (183 SWE-Gym tasks) with identical tokens, rewards, loss masks and 81,200 optimizer steps (Table 7, ESS >= 0.993). Every checkpoint is graded by a sealed per-instance SWE-bench Verified oracle on the four source harnesses (500 tasks, avg@2, 24,000 episodes, 99.1% verdict coverage) and on a minimal weak-ReAct harness held out of training (k=4/k=8). Claims: (1) the evaluation harness dominates, 4.3x vs 1.16x for the recipe; (2) Cross adds no detectable portable gain; (3) Cross's advantage carries recoverable harness identity with no downstream effect; (4) both rules' gains concentrate on source harnesses - configuration adaptation, not portable capability.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the difference between Cross and Within grouping in multi-harness RL?
In GRPO-style agent RL, a group of trajectories for the same task is compared against each other to form advantages. 'Within' forms one group per task-harness pair, so each coding harness (e.g., Aider, SWE-agent) keeps its own baseline. 'Cross' pools all harnesses' trajectories for a task into a single group, so outcomes from different harnesses compete for the same credit. The paper finds this choice changes what the advantage signal contains but not the policy's portable performance.
Does pooling harnesses in GRPO help an agent generalize to unseen harnesses?
In this study, no detectable effect. On a minimal ReAct harness held out of training, Cross minus Within was +0.25 pp [-0.48, +1.02] at eight attempts per task and +0.16 [-0.41, +0.72] pooled over three training seeds, with individual seed estimates changing sign. The design could only detect effects of roughly 1 percentage point or larger, so smaller true benefits cannot be ruled out.
How much does the choice of agent harness change SWE-bench scores?
Substantially. In this paper's controlled matrix over 24,000 sealed evaluations of the same checkpoints, mean solve rate ranged from 2.14% (Aider) to 9.27% (OpenHands) - a factor of 4.3 - while the training recipe changed it by only a factor of 1.16. A second model family (Seed-Coder-8B) showed different harness effects, so harness strength is not a fixed scalar that can be subtracted from every model.
Is the paper's null result just because the experiment was underpowered?
Partly a matter of resolution, and the authors say so. The smallest detectable difference is 1.19-1.42 pp at four attempts per task and 0.88-1.14 pp at eight; three-seed pooling narrows this to about 0.8 pp. The null therefore bounds any portable Cross benefit at roughly under 1 pp, which is far below the +9.5 to +22.6 pp gains some industrial multi-harness systems report, so the result neither confirms nor contradicts those systems.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded cell values (Figures 2-4) are extracted as run-on caption text; every number cited in this review was cross-checked against Tables 1-5, 7-10 and Section 6 text, so no cited value relies on garbled figure text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.