CHART: A Harness-Rotation Curriculum for Harness-Robust Search Agents
Xinlu Zhang, Ying-Chun Lin, Zhihan Zhang, Besnik Fetahu, Xi Chen
cs.LG · cs.AI
Quick verdict
Claims harness-local RL behavior and proposes a rotation curriculum (CHART) fixing it; results well-ablated but single-run, single-model, no code.
Summary
The paper studies a fragility of RL-trained search agents: behavior learned under one harness (system prompt) disappears when it is reworded. Training Qwen3-30B-A3B with GRPO to parallelize web searches, the authors find the behavior is harness-local — 91% parallel turns under the training harness but 2% on cue-free ones — and static harness augmentation fails at every pool size (1–33), because small pools exhaust the parallel-vs-serial reward contrast while large pools dilute it. They propose CHART, a curriculum keeping a small rotating window of harnesses: every 20 steps, harnesses whose parallel rate reaches 0.85 graduate and mid-band (0.40–0.80) candidates replace them. From the same 33-harness pool, CHART-3 reaches 100% in-pool coverage, 89% parallel turns on 12 held-out cue-free harnesses (static pools ≤5%), and +5.6pp pass@1 on an out-of-domain shopping task; a CHART-trained model is also a better substrate for meta-harness search (+12.6pp over untrained on deep-research QA). Ablations cover rotation length, refill rule, and graduation trigger. Limitations: one behavior, one base model, system-prompt-only variation, single training runs without significance tests, and no code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- RL under a fixed harness installs learned behavior as harness-local: a model parallelizing on 91% of turns under its training harness drops to 2% parallel turns on cue-free harnesses.
- Static harness augmentation fails at every pool size (1–33) because small pools exhaust and large pools dilute GRPO's within-group parallel-vs-serial reward contrast.
- CHART's rotating window, from the same 33-harness pool, yields 100% in-pool coverage, 89% held-out parallel turns (static pools ≤5%), and +5.6pp pass@1 on an OOD shopping task.
- A CHART-trained model is a strictly better substrate for meta-harness search, +12.6pp over the untrained base on deep-research QA best-of-search.
- Rotation must run throughout training, not merely as an early warm-up phase (early-stop covers 14/33 harnesses vs full-run 23/33, Table 6).
Strengths
Novel problem framing with a falsifiable diagnostic
The cue-vs-cue-free PR% split (groups A/C vs B/D, §3.1/§4, Appendix A) cleanly separates instruction-following from internalization; harness-locality itself (91% → 45%/2% PR%, §5.1) is a genuinely new failure mode for agentic RL, positioned against prompt-sensitivity literature for frozen LLMs (§2).
Controlled single-variable design with premise verification
Every fixed-N/CHART-N pair is seeded from the same N harnesses under a matched 200-step budget (§5); Appendix B's budget-matched sequential baseline independently verifies the premise (parallel 0.513 vs sequential 0.473 pass@1, 26.5k vs 67.0k input tokens); Appendix C validates the 50-datapoint graduation eval (97% decision agreement with the full eval).
Complete ablations including honest negative results
Confound-settling model×harness cross
Transparent failure reporting
Weaknesses
Single run per configuration, no statistical testing
Narrow external validity
Self-authored evaluation distribution; harshest test excluded
Reproducibility blockers
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper makes three linked claims. First, a new failure mode: RL-learned agentic behavior is *harness-local*. A Qwen3-30B-A3B model GRPO-trained to parallelize web searches issues parallel calls on 91% of turns under its own training harness, but only 45% on other parallel-requesting harnesses and 2% on cue-free ones (§5.1) — the capability survives; its elicitation does not. Second, static harness augmentation fails at *every* pool size (1–33): fixed pools never lift cue-free parallelism past 13% nor cover more than 16/33 harnesses (Table 2), because GRPO's learning signal is the within-group parallel-vs-serial reward contrast, which small pools saturate and large pools dilute. Third, CHART — graduate harnesses (PR% ≥ 0.85, checked every 20 steps) and refill from a learnable mid-band (0.40–0.80) — fixes this from the *same* pool: 100% in-pool coverage (vs ≤48% static), 89% parallel turns on 12 unseen cue-free held-out harnesses (vs ≤5% static), +5.6pp pass@1 on the OOD shopping task (0.437 vs best fixed 0.381), and a better substrate for meta-harness search: +12.6pp over the untrained base on deep-research QA best-of-search (0.535 vs 0.409, Table 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'harness-local' mean in this paper?
A behavior learned by RL — here, issuing multiple web searches in parallel within one turn — that appears only when the agent runs behind the same system prompt used during training. Rewording the prompt makes the model revert: 91% of turns are parallel under its training harness, but only 2% on cue-free prompts.
What is CHART?
Curriculum HArness Rotation Training: during GRPO training, the agent sees only a small rotating window of system-prompt harnesses. Every 20 steps, harnesses whose parallel rate reaches 0.85 'graduate' and are replaced by mid-band candidates (parallel rate 0.40–0.80), keeping the parallel-vs-serial reward contrast alive throughout training.
Why doesn't training on many prompt variants fix prompt overfitting?
In this GRPO setup, learning comes from the reward gap between parallel and serial rollouts of the same question. Small harness pools saturate that gap (everything gets covered), while large pools split the rollout budget so thinly that no harness consolidates; measured across pool sizes 1–33, static pools never exceed 13% cue-free parallelism.
What results does CHART achieve?
From the same 33-harness pool, CHART-3 covers 100% of pool harnesses, parallelizes on 89% of turns on 12 unseen cue-free harnesses (static pools ≤5%), improves pass@1 by 5.6pp on an out-of-domain shopping task, and makes the model a better substrate for automatic harness search (+12.6pp on deep-research QA vs the untrained base).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract as supplied in the task metadata cuts off mid-sentence, but the complete abstract appears at the start of the full-text extraction — no content was lost.
- The provided text ends after Appendix D (page 13) with a complete closing sentence; any material beyond Appendix D was not provided and its existence cannot be confirmed from the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.