Reach or Solve? Attributing Agentic RL Gains with Checkpoint Handoffs
cs.AI
Quick verdict
Checkpoint handoff splits agentic RL gains into reach and solve; solid protocol, positive interaction in all 5 conditions, but small samples, Qwen-only.
Summary
The paper addresses the endogenous state problem in evaluating agentic RL: an agent's early actions determine the states it is later scored from, so endpoint success conflates state visitation with state-conditional competence, and conditioning only on reached states introduces selection bias. The authors introduce checkpoint handoff, a protocol that clones an environment-verified state reached by one checkpoint (the reacher) and hands it to another (the solver) without retraining. Crossing SFT and RL checkpoints in both roles yields a 2x2 factorial whose interaction splits endpoint gains into REACH (arrival at a frontier of states exactly two actions from success) and SOLVE (completion from identical cloned states). Experiments span TravelPlanner with three Agent-STAR SFT/RL pairs (1.5B-7B) and ALFWorld with a SkillRL 7B pair. The reacher-by-solver interaction is positive in all five conditions (e.g., +24.2 pp [13.3, 35.0] on ALFWorld unseen). On ALFWorld, RL reaches the frontier about six times as often (82.5% vs 13.3%) and completes 95.7% vs 56.5% from identical states; independent REACH and SOLVE gaps predict the interaction within 3.5 pp. Limitations: small task samples (30 ALFWorld tasks, 115 frontier states, SFT arrivals from six tasks), Qwen-family-only checkpoints, and a TravelPlanner handoff rule that cuts late in trajectories.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Endpoint success comparisons between SFT and RL agent checkpoints conflate state visitation with state-conditional value, and conditioning on reached states flips the sign of the measured solver effect in their data.
- Checkpoint handoff separates REACH from SOLVE on released checkpoints without retraining, using an environment-verified frontier and exact cloned-state continuation.
- The reacher-by-solver interaction is positive in all five tested conditions, meaning RL solvers gain more on RL-reached states than on SFT-reached ones.
- On ALFWorld, RL improves both components: it reaches the F2 frontier about six times as often (82.5% vs 13.3%) and completes far more often from identical cloned states (95.7% vs 56.5%).
- Independently measured REACH and SOLVE gaps predict the observed interaction to within 3.5 percentage points (20.7 predicted vs 24.2 observed).
Strengths
Rigorous estimand framing
Sections 2.1-2.4 formalize the problem with potential outcomes (Rubin) and a 2x2 factorial interaction (Eq. 3-4), and prove conceptually why endpoint comparison conflates two factors and why reached-only conditioning changes the estimand; Appendix C quantifies the latter (survivor-only -12.6 pp [ -45.8, 22.7] vs endpoint +24.2 pp).
Environment-verified frontier and clone audits
F2 membership (states exactly two valid actions from success) is certified by exhaustive replay enumeration - 3,227 first and 89,511 second actions across 115 states, never consulting the checkpoints; clones are verified by reset-and-replay equality on observation, admissible actions, prompt hash and token digest (Appendix A).
Consistent effects with honest uncertainty
Reproducibility hygiene
Weaknesses
Thin ALFWorld samples in the cells that matter
Interaction mechanism not fully identified
Decomposition evidence rests on one pipeline
Narrow architecture and unresolved scale trend
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a clean identification problem: in closed-loop agent evaluation, an SFT checkpoint and its RL counterpart are scored from states each produced itself, so endpoint success confounds state visitation with state-conditional value (the endogenous state problem, Section 2.1). The authors formalize this with potential outcomes and a 2x2 reacher-by-solver factorial (Equations 3-4), define an environment-verified solvable frontier F_D of states exactly D=2 valid actions from success with sufficient budget (Equations 5-6), and propose checkpoint handoff: replay a reacher to a prespecified cut, clone state/history/budget, and let both solvers continue without retraining. Central claims: (1) endpoint comparison cannot attribute the RL gain and reached-only conditioning flips its sign in their data; (2) the interaction I_tau is positive in all five conditions; (3) on ALFWorld RL improves both REACH and SOLVE; (4) independent component gaps predict the interaction within 3.5 pp.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the endogenous state problem in LLM agent evaluation?
Because a language-model agent's own earlier actions determine which situations it later faces, an SFT checkpoint and an RL checkpoint end up being scored from different states even on identical tasks. Endpoint success then mixes two things: how often the agent arrives at promising states and how well it acts once there. This paper further shows that restricting comparisons to states both policies reach makes things worse - in their data it flips the sign of the measured effect.
How does the checkpoint handoff protocol work?
One checkpoint (the reacher) acts on a task until it reaches a state the environment verifies is exactly two valid actions from success. That state - environment, history, and remaining budget - is cloned and handed to another checkpoint (the solver), which continues live with no retraining. Crossing SFT/RL checkpoints in the two roles yields a 2x2 design whose interaction splits the endpoint gain into arrival (REACH) and completion (SOLVE).
Does RL actually make agents better decision-makers according to this paper?
The paper argues RL gains come from both components it separates. On ALFWorld, the RL checkpoint reached a verifiably solvable state about six times as often as SFT (82.5% vs 13.3%), and it also completed far more often from byte-identical cloned states (95.7% vs 56.5%). So RL both reaches better states and acts better from the states it reaches.
Which benchmarks and models were used in the study?
The authors evaluated publicly released Qwen-family checkpoints without retraining: an SkillRL 7B SFT/RL pair on ALFWorld (household tasks) and three Agent-STAR SFT/RL pairs at 1.5B, 3B, and 7B on TravelPlanner (trip planning), using fixed seeds, pinned model revisions, and the environments' native success flags for scoring.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-10 are available only through captions and in-text numbers; visual details were not verifiable, though all headline values appear in Tables 1-3 and 6-7.
- Text extraction may contain minor OCR artifacts; the arXiv identifier and 2026-era citations could not be verified offline.
- The anonymous supplementary archive referenced in the reproducibility statement could not be inspected; no code repository is mentioned, so reproducibility rests on the protocol description and public checkpoints/environments.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.