Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents
Liming Pu, Xiaoxia Li, Yifu Liu, Teng Cao, Bin Yang
cs.LG · cs.AI
Quick verdict
Claims outcome-only RL suffices for long-horizon agents via large groups and KL-anchored on-policy updates; strong results but key ablations confounded.
Summary
The paper argues that the reported ceiling of outcome-only RL for long-horizon LLM agents stems from two fixable failures of practice: signal starvation (group-relative advantages vanish when rollout groups contain all-success or all-fail outcomes; with prior group sizes n≤8 the hardest tasks are almost always silent, quantified by Psig(p,n)=1−p^n−(1−p)^n) and policy drift (entropy collapse when many updates are squeezed from a small task pool). The CANOPY protocol answers with large same-task groups (n=32), strictly on-policy single updates per rollout batch (measured clip fraction and PPO-KL identically zero across training), a KL anchor to the base model (β=1e-4), a pooled token-mean loss over action tokens only, a sparse 0/1 reward, and fault quarantine for exogenous environment failures; at test time the turn/context budget is enlarged. A Qwen3-14B policy reaches 86.9 Test-Normal and 67.6 Test-Challenge TGC (mean@1, AppWorld leaderboard, Feb 2026) versus base 32.4 mean@4, and lifts Qwen3.5-9B by +16.6 on SWE-bench Verified (31.3→47.9 mean@4). Six ablations order as the theory predicts. Limitations: one training run per ablation without seed variance; the n=8 comparison confounds coverage with total sampling; environment stabilization alone moved scores from ~47 to 73–74; and two training-free systems on closed frontier backbones report higher scores.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The reported ceiling of outcome-only RL on small open agents is an artifact of signal starvation (undersized GRPO groups) and policy drift, not a limit of RL itself.
- A Qwen3-14B policy trained with CANOPY reaches 86.9 Test-Normal / 67.6 Test-Challenge TGC (mean@1) and was the top trained-policy AppWorld entry as of Feb 2026.
- RL expands beyond what base-model resampling reaches: a single run at 86.9 exceeds the base model's best@4 of 58.9 (67.6 vs 37.7 on Test-Challenge).
- Each CANOPY ingredient is priced by ablations whose ordering matches the two-failure diagnosis (n=8 −16.4, off-policy −17.4, KL −7.0, drop-hardest −6.0, token-loss −5.4, dense reward −1.8).
- The same design principles transfer to SWE-bench Verified with repository-level decontamination, lifting Qwen3.5-9B by +16.6 mean@4 (31.3→47.9).
Strengths
Diagnosis is formalized and empirically matched
The coverage formula Psig(p,n)=1−p^n−(1−p)^n (Eq. 4, Fig. 2) is validated against rollout logs: all-fail groups vanish within ~10 steps, and the hardest tier L3 remains the last informative gradient source late in training (Fig. 3B), exactly the saturation regime the equation predicts.
The strictly-on-policy claim is verified, not asserted
Appendix C reports actor/pg_clipfrac and actor/ppo_kl identically zero across all 90 steps of the main run, so the clipped surrogate demonstrably reduces to plain policy gradient; the off-policy contrast variant logs nonzero clip fraction from step 1.
Unusually candid limitations sections
Fault quarantine before scoring
Reproducibility documentation is exceptional
Weaknesses
Single training run per ablation, no seed variance
Central coverage claim is confounded
Infrastructure confound of ~26 TGC points
Leaderboard framing overstates in the abstract
Cross-method comparisons conflate backbone
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a sharply framed question: is the widely reported ceiling of outcome-only RL on small open LLM agents real, or an artifact of training practice? The authors argue the latter, identifying two mechanisms. Signal starvation: under GRPO-style group-relative advantages, a rollout group with all-success or all-fail outcomes contributes zero gradient, and at the group sizes prior AppWorld RL used (n≤8) the hardest tasks are almost always silent — quantified by Psig(p,n)=1−p^n−(1−p)^n (§3.2, Figure 2). Policy drift: entropy collapse when many updates are extracted from a small, repeatedly revisited 90-task pool, traced to four named causes (§3.3). The CANOPY protocol pairs each cause with one ingredient: same-task groups sized from a pilot coverage estimate (n=32), strictly on-policy updates (gradient batch = rollout batch, one pass), a KL anchor to the fixed base model (β=10−4, k3 estimator), a pooled token-mean loss over action tokens only, a sparse {0,1} reward, and fault quarantine of exogenous environment failures before scoring; test-time budget transfer (50 turns/32k → 100 turns/61k) completes the recipe. Headline claims: Qwen3-14B reaches Test-Normal 86.9 / Test-Challenge 67.6 TGC (mean@1, Feb 2026), framed as topping the leaderboard; the base model's mean@4 rises from 32.4 to 83.2 (Table 3); the same principles lift Qwen3.5-9B by +16.6 on SWE-bench Verified (31.3→47.9 mean@4, Table 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CANOPY?
CANOPY (Coverage-ANchored On-PolicY RL) is a minimalist recipe for post-training LLM agents with outcome-only rewards: large same-task rollout groups (n=32) so hard tasks produce learning signal, strictly on-policy updates with one gradient step per rollout batch, a KL anchor to the base model, a token-level loss over action tokens only, and an enlarged turn/context budget at test time.
How well does it perform on AppWorld?
A Qwen3-14B policy trained with CANOPY scores 86.9 TGC on Test-Normal and 67.6 on Test-Challenge (mean@1, official leaderboard, Feb 2026) — the top entry among trained policies. Two training-free systems built on closed frontier models report higher numbers (HCL-GP 98.2 under a non-standard protocol; ASSAY 89.3).
Does CANOPY need dense or step-level rewards?
No. It uses a sparse 0/1 reward that credits only trajectories passing all unit tests. In ablation, a dense partial-credit variant scored just 1.8 points lower, which the authors read as evidence that dense rewards were compensating for under-exploration rather than being necessary.
How much compute does the training require?
The AppWorld run consumed about 2,690 GPU-hours (8 nodes × 8 GPUs for 42 hours; 259,200 trajectories, 3.47×10^9 tokens). 'Minimalist' in the paper refers to the number of algorithmic components, not to compute cost.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The metadata-supplied abstract is cut mid-sentence, but the complete abstract appears at the start of the embedded full text, so no analysis gap results.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.