arXiv paper2026-09-21· machine learning — reinforcement learning for LLM agents· method paper

Luck Is Not Skill: When Do Paired Rollouts Help Group-Relative RL of LLM Agents?

Q: What are paired rollouts in GRPO-style RL?

All rollouts of the same prompt share one environment noise schedule — the same draws decide which tool calls fail and whether the grader flips — implemented with event-keyed hashing (a common-random-numbers technique). Each rollout's marginal environment distribution stays the same; only the dependence among siblings changes, so within-group reward comparisons are less contaminated by luck.

Q: Does pairing actually improve learning of LLM agents?

In this paper's pre-registered study, the variance mechanism holds (lower gradient-covariance traces at all eight probed checkpoints), but both registered primary learning criteria missed. Under tool faults, final test success favored pairing by +5.1 points with three positive seeds; under grader flips the effect was indistinguishable from zero (+0.003 AUC). The paper supports the variance mechanism, not a general learning benefit.

Q: When does pairing reduce gradient variance?

The paper derives an exact condition for one-sided grader noise (Equation 5): pairing lowers the gradient-variance trace only if the squared size of the clean group gradient is smaller than the score spread of the truly successful rollouts. A counterexample shows pairing can reduce reward-contrast variance while increasing gradient variance 2.2-fold, so there is no general guarantee.

Q: Is the code and data available?

The paper states that code, task pools, the timestamped pre-registration, and the run/decision/probe registers are publicly available at github.com/TheDeadcoder/paired-rollouts. Raw rollouts and trainer checkpoints are not released, so full independent re-execution of training and the gradient probe is not possible from the repository alone.

Nazmus Sakib

View original paper ↗arXiv:2609.24144

cs.LG

Quick verdict

Paired rollouts in group-relative RL: exact theory and unusually honest evidence; mechanism supported, but both registered learning criteria missed.

Summary

The paper studies paired rollouts in GRPO-style group-relative RL for LLM agents: all rollouts of a prompt share one event-keyed environment-noise schedule (common random numbers) while each rollout's marginal distribution is preserved. Theoretically, pairing removes the between-schedule component of reward-contrast variance (factor 1−λ, the 'luck share') but need not reduce gradient variance; for one-sided grader noise the author derives an exact condition (Eq. 5) and gives an 8-arm bandit counterexample where reward contrasts improve yet gradient variance rises 2.2-fold. Empirically, a pre-registered study (frozen 2026-09-08, RFC-3161 timestamped, with a disclosed completed pilot) trains a Qwen3.5-2B LoRA tool-use agent in a synthetic back-office simulator under tool faults (p=0.25) and grader flips (q=0.10), three seeds per arm. Mechanism checks pass: spurious-variance rates match theory (55.8% vs 0%), and a gradient probe at eight checkpoints finds paired covariance-trace reductions of 21–30% (grader) and 40–63% (tool faults). However, both registered primary learning criteria miss: validation-AUC differences are +0.037 (tool faults, one negative seed) and +0.003 (grader flips, CI [−0.029,+0.033]). Final noisy-test success favors pairing by +5.1pp under tool faults, but with only three seeds and a recovery interval spanning zero. Limitations: one model, one synthetic environment, 100 steps, pre-freeze test-pool exposure, and cross-stack hardware heterogeneity.

Review complete

Assessment at a glance

machine learning — reinforcement learning for LLM agents

risk flags identified

● high 0

● medium 3

● low 1

Reproducibility readinessmedium

Code & data availableyes

Review confidencehigh

AI text signal0/20 passages flagged

Deep research viableCode-reproducibleNeeds GPU

4 strengths · 4 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

Pairing (one shared event-keyed noise schedule per group) removes exactly the between-schedule share of reward-contrast variance (factor 1−λ) while preserving each rollout's marginal environment law and the unbiasedness of on-policy leave-one-out estimators.
Pairing has no general gradient-variance guarantee: under one-sided grader noise it reduces trace iff E‖gc‖² < (1/G²)EΣᵢ rᵢ‖Sᵢ−S̄‖² (Eq. 5), and an enumerated bandit shows reward contrasts improving while gradient variance rises 2.2-fold.
In a pre-registered 2B-agent study the mechanism passes (spurious rates 55.8% vs 0%; covariance-trace reductions of 21–30% under grader flips and 40–63% under tool faults at all eight probed checkpoints), but both registered primary learning criteria miss.
Under tool faults, final noisy-test success favors pairing by +5.1pp on average with all three seed differences positive, though the registered AUC criterion fails and the recovery interval includes zero.

Strengths

Exact, falsifiable theory separating three distinct effects
Section 4 distinguishes reward-contrast variance (Prop 2, factor 1−λ), gradient variance (Prop 3, closed-form iff condition Eq. 5), and learning; the fully enumerated 8-arm bandit (Appendix A.1, reward-contrast −9% but gradient variance 0.002615→0.005845) is a clean counterexample and doubles as a unit test of the probe code.
Unusually disciplined pre-registration with unfavorable results honored
Protocol frozen 2026-09-08 with two RFC-3161 timestamps, disclosed completed pilot, logged amendment A1 and deviations (Section 5.5, Appendix D); both primary H2 criteria are reported as missed and the paper adopts the registered gate-2 boundary framing rather than post-hoc storytelling.
Mechanism checks that quantitatively match theory
Probe engineering rigor

Weaknesses

Both registered primary learning criteria missed
Cross-stack confound larger than the measured effect
Probe lacks uncertainty quantification and coverage
Weak external validity and confirmatory exposure

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

The paper asks whether pairing the environment randomness of rollout groups — all G rollouts of a prompt share one event-keyed noise schedule (common random numbers; event-keyed hashing after Buffalo et al., 2026) — improves group-relative RL (GRPO-style) of LLM agents. Three distinct claims are cleanly separated: (i) pairing improves within-group reward contrasts (Prop 2: contrast variance falls by exactly 1−λ, where λ is the 'luck share' σ²env/Var(R)); (ii) pairing reduces gradient variance — shown false in general, with an exact iff condition under one-sided grader noise (Prop 3, Eq. 5) and an enumerated 8-arm bandit counterexample where contrast variance falls 9% while gradient variance rises 2.2-fold (0.002615→0.005845, Appendix A.1); (iii) pairing improves learning — tested in a pre-registered controlled study and, by the paper's own registered rules, not established. The abstract does not overclaim beyond what Sections 6.2–6.4 support.

Methodology Assessment

The theory (Sections 3–4) is precise: unbiasedness of the leave-one-out estimator survives coupling (Prop 1, with a careful remark that the standardized estimator's expectation can differ across designs); Prop 3's closed-form condition is the central contribution, illustrated with orthogonal scores (pairing helps iff k ≥ 2 true successes) and the bandit. The empirical apparatus is unusually disciplined: an RFC-3161-timestamped frozen protocol (2026-09-08), a disclosed completed pilot, logged amendment A1 and deviations (Section 5.5, Appendix D), matched seed pairs on

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 4 strengths & 4 weaknesses, analyzed in detail
4 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What are paired rollouts in GRPO-style RL?

All rollouts of the same prompt share one environment noise schedule — the same draws decide which tool calls fail and whether the grader flips — implemented with event-keyed hashing (a common-random-numbers technique). Each rollout's marginal environment distribution stays the same; only the dependence among siblings changes, so within-group reward comparisons are less contaminated by luck.

Does pairing actually improve learning of LLM agents?

In this paper's pre-registered study, the variance mechanism holds (lower gradient-covariance traces at all eight probed checkpoints), but both registered primary learning criteria missed. Under tool faults, final test success favored pairing by +5.1 points with three positive seeds; under grader flips the effect was indistinguishable from zero (+0.003 AUC). The paper supports the variance mechanism, not a general learning benefit.

When does pairing reduce gradient variance?

The paper derives an exact condition for one-sided grader noise (Equation 5): pairing lowers the gradient-variance trace only if the squared size of the clean group gradient is smaller than the score spread of the truly successful rollouts. A counterexample shows pairing can reduce reward-contrast variance while increasing gradient variance 2.2-fold, so there is no general guarantee.

Is the code and data available?

The paper states that code, task pools, the timestamped pre-registration, and the run/decision/probe registers are publicly available at github.com/TheDeadcoder/paired-rollouts. Raw rollouts and trainer checkpoints are not released, so full independent re-execution of training and the gradient probe is not possible from the repository alone.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Figures 2–5 are embedded only as text labels; curve shapes and spread beyond the quoted numbers could not be visually inspected.
The GitHub repository, RFC-3161 tokens and registers are asserted in the text but not independently verified in this review.
Tables were flattened by text extraction; row/column alignment was cross-checked against surrounding prose, but minor formatting loss is possible.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper