HarnessBandit: Joint Learnability-Transferability Scheduling for Multi-Harness Agentic Reinforcement Learning
Hongliang Wei, Xiaobing Tu, Yinggui Wang, Zhengxi Liu, Rongkun Xue, Jinkui Ren, Xiantao Zhang, Debin Zhao, et al.
cs.LG · cs.AI · cs.CL
Quick verdict
Bandit scheduler picks the training harness per GRPO step; gains over mixed batches look plausible but come from single small runs without signal ablations.
Summary
The paper targets robustness of language-model agents to harness variation — differing system prompts, tool schemas, and control loops — by casting multi-harness RL fine-tuning as a non-stationary multi-armed bandit. At each optimizer step, HarnessBandit selects one harness for a harness-pure rollout batch. From the unchanged GRPO step it extracts two signals: learnability (mean absolute advantage on the batch) and transferability (cosine between a 65,536-dimensional proportional coordinate subsample of the current harness's last-four-layer gradient and exponential moving averages of the other harnesses' sketches). The signals are min-max normalized over a pooled 32-step sliding window, fused 50/50 into an EMA value, and sampled with a UCB-style visit bonus, Boltzmann temperature, and a 15% uniform exploration floor after a 30-step round-robin warmup. Training Qwen3.5-2B for 150 GRPO steps on 2,000 ClawGym tasks across six harnesses yields PinchBench 0.506 versus 0.493 for a two-harness mixture, 0.331 for a six-harness mixture, and 0.330 for the base model; on ClawEval it reaches 41.6% pass@3 and 0.556 mean score versus 36.0%/0.534 for the base. Limitations: single training runs without seeds or confidence intervals, no learnability-only or transferability-only ablation, an unexplained zero writing score for the six-harness mixture, and no stated code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Multi-harness agentic RL can be cast as a non-stationary bandit over execution interfaces, with learnability (mean absolute GRPO advantage) and cross-harness transferability (sketched-gradient cosine) as trainer-native arm utilities (§3, §4).
- A UCB-softmax scheduler fusing the two signals beats static mixed-batch training at matched GRPO budget: PinchBench 0.506 vs 0.493 (two-harness mix) and 0.331 (six-harness mix) (Table 1).
- On ClawEval (held-out tasks and harness), HarnessBandit matches the untuned base model on strict Pass3 (43/161, 26.7%) while exceeding it on pass@3 (41.6% vs 36.0%) and mean score (0.556 vs 0.534), and beats both mixtures on all three metrics (§5.3).
- Learnability and transferability are distinct, non-monotone, evolving signals, and the induced schedule is a mild reallocation (probabilities 0.127-0.204) rather than arm collapse (Fig. 2, App. C/F).
Strengths
Novel problem formulation at the interface level
§2-3 position the contribution as scheduling execution interfaces (harnesses wrapping the same ClawGym tasks) rather than tasks or domains, cleanly distinguished from multi-task RL (Caruana, Yu et al.) and curriculum/data-mixing work; the paper itself notes the closest prior is a two-signal bandit for reasoning domains (Yang et al., 2026), and states the difference honestly.
Trainer-native, cheap signals with bounded overhead
§4.3/§4.6: both signals come free from the existing GRPO step; the 65,536-dim proportional coordinate subsample of the last K=4 layers keeps the EMA store at megabytes instead of 190M-dim buffers, and overhead is one gather, one d-vector all-reduce, and O(Hd) cosines.
Honest and well-designed evaluation split
Unusually transparent diagnostics
Weaknesses
No statistical uncertainty on the headline claim
The two-signal fusion is never ablated
Strict ClawEval metric only ties the base model
Unexplained degenerate baseline cell
Nine fixed scheduler hyperparameters, no sensitivity analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses a real and timely problem: when one policy is RL-trained through several agent harnesses (system prompt + tool schema + control loop), which harness should generate each on-policy batch? The authors cast this as a non-stationary multi-armed bandit (§3) whose arms are the six harnesses, with two trainer-native utilities: learnability L (mean absolute GRPO advantage, Eq. 2/4) and transferability T (cosine between a 65,536-dim proportional coordinate subsample of the current harness's last-four-layer gradient and per-harness sketch EMAs, Eq. 3/5). Signals are pooled-window min-max normalized (W=32), fused 50/50 into an EMA value (β=0.3), and sampled via UCB-softmax (τ=0.3, c/√(n+1) bonus, ε=0.15 floor) after a 30-step round-robin warmup (§4). Three claims are made: the bandit formulation is new at the interface level; the two signals are complementary and evolving; and the scheduler beats static mixtures at a matched 150-step GRPO budget.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a harness in HarnessBandit?
A harness is the agent execution interface wrapping a task: the system prompt, tool schema, planner/control loop, and trajectory format. The paper trains one policy through six harnesses (Gemini CLI, OpenCode, Qwen Code, Pi, OpenClaw, Codex) and schedules which one generates each training batch.
How much does HarnessBandit improve over mixed-batch training?
At a matched 150-step GRPO budget on Qwen3.5-2B, it scores 0.506 on PinchBench versus 0.493 for a two-harness OpenClaw+Codex mixture and 0.331 for a six-harness mixture (untuned base 0.330). On ClawEval it reaches 41.6% pass@3 and 0.556 mean score, versus 38.5%/0.510 and 29.8%/0.490 for the two mixtures; its strict Pass3 rate (26.7%) only ties the base model.
What model, data, and compute were used?
Qwen3.5-2B trained with Dr.GRPO on 2,000 ClawGym tasks crossed with six harnesses (12,000 prompt rows), batch size 8 with 8 samples per prompt, for 150 optimizer steps on four GPUs. Evaluation uses held-out PinchBench (147 tasks) and ClawEval (161 tasks, 3 trials each).
Is code available for HarnessBandit?
The paper does not state a code release for the scheduler itself. It builds on public components — the Polar RL stack, slime/Megatron GRPO, the ClawGym task suite, and the PinchBench and ClawEval benchmarks — but the bandit scheduler implementation is not described as released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction is complete through Appendix F and the references; nothing appears truncated. Figure E.1's embedded axis/legend text is garbled by PDF extraction, though its caption conveys the panel structure.
- Provided metadata lists eight authors; the extracted byline lists nine, adding Xiaopeng Fan as a co-corresponding author.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.