Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL
Juzheng Zhang, Disha Makhija, Manoj Ghuhan Arivazhagan, Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah
cs.LG · cs.AI · cs.CL
Quick verdict
Claims unmasking observations in agent SFT improves later RL pass@k; well-controlled, mechanistic evidence, but single-run results leave margins within noise.
Summary
The paper tests whether the standard SFT convention of applying loss only to agent action tokens is the best initialization for downstream RL. It introduces ActObs, a one-line change that also applies the language-modeling loss to observation tokens already present in each trajectory (equal weight, λ=1), with no extra data, parameters, tokens, or forward passes. Qwen3-4B/8B models are SFT'd on 50k synthetic terminal trajectories (0.71B tokens, DeepSeek-V3.2 teacher) and then trained with an identical GRPO recipe. After SFT, all initializations perform similarly on Terminal-Bench 2.0; after GRPO they diverge: at 4B, ActObs→GRPO beats action-only SFT at every sampling budget (7.2 vs 5.6 pass@1, a 29% relative gain); at 8B, it trades roughly 1.3 pass@1 points for +3.4 pass@16 points and 24 vs 21 solved tasks. The advantage transfers to the unseen code-editing benchmark aider-polyglot (+4.2 pass@1 at 4B), and joint supervision beats a sequential observation-then-action control. Mechanistically, action and observation gradients become orthogonal within roughly 10-20 SFT steps; action-only training leaves a large residual observation gradient and degrades environment prediction below the base model. Key limitations: one training run per configuration (bootstrap errors capture only evaluation noise), no multiple-comparison control across the large results grid, scope limited to terminal agents and one model family, and code not yet released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Unmasking observation tokens during agent-trajectory SFT leaves immediate benchmark performance nearly unchanged but yields a substantially stronger initialization for GRPO.
- At 4B, GRPO from ActObs achieves higher pass@k at every evaluated budget on Terminal-Bench 2.0 (+1.6pp pass@1, 29% relative); at 8B it trades ~1.3pp pass@1 for +3.4pp pass@16 and 24 vs 21 solved tasks.
- The advantage transfers cross-domain to unseen aider-polyglot code editing (+4.2pp pass@1 at 4B), and joint supervision is necessary: the sequential Obs→Act control does not reproduce it.
- Action and observation gradients become orthogonal within 10-20 SFT steps; action-only training leaves a ~41x residual observation gradient and degrades environment prediction below the base model, while ActObs prevents this one-sided specialization.
- ActObs-initialized policies retain more entropy during RL with less policy movement from initialization, and the high-k advantage persists when ActionSFT's inference temperature is raised to match its entropy.
Strengths
Minimal, perfectly controlled intervention
ActObs changes only the loss mask (Eq. 1): same corpus, tokens, optimizer, and forward passes (Sec. 2). The ActionSFT-vs-ActObs comparison is therefore a clean isolation of the supervision target, with denominator normalization keeping per-example loss scale fixed as λ varies.
Thoughtful ablation ladder
Obs→Act timing control (Sec. 3.2) separates joint from sequential supervision; shuffled obs collapse (8B pass@1 7.9→3.7) vs wild logs retaining 7.9 (Table 2); act2act and ActObs-100 rule out training-amount explanations; the λ sweep (Table 6) shows a graded pass@1→pass@8 tradeoff; temperature matching (Sec. 4.2, T*=0.64) rules out inference-time randomness as the source of the pass@k gain.
Mechanistic analysis beyond score-chasing
Honest reporting of tradeoffs
Cross-domain and cross-objective robustness checks
Weaknesses
Single training run per configuration
No multiple-comparison control
Narrow external validity
Scaffold non-parity is a live confound
Nothing independently verifiable today
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a specific, well-chosen convention: agent-trajectory SFT applies the LM loss only to assistant-authored action tokens (citing Zeng et al. 2024, Chen et al. 2023/2024), leaving environment observations as context. ActObs (Eq. 1) unmasks those observation tokens at equal weight λ=1, changing nothing else — same corpus, sequence, optimizer, and forward passes. The central claim is not that ActObs is a better SFT objective by benchmark score, but that it is a better *initialization for GRPO*: SFT-time scores are near-identical, post-RL policies diverge. Supporting claims: (i) at 4B, higher pass@k at every budget post-GRPO; (ii) at 8B, a pass@1-for-pass@k trade (+3.4pp at pass@16, 24 vs 21 tasks); (iii) cross-domain transfer to aider-polyglot; (iv) a mechanistic account via gradient orthogonality (Eq. 2, Fig. 5) and the entropy/policy-movement analysis of Sec. 4 and Table 7.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ActObs in one sentence?
ActObs is a one-line change to supervised fine-tuning for LLM agents: instead of masking environment observation tokens out of the loss, it trains the model to predict them too, using observation tokens already present in each trajectory — no new data, parameters, sequence tokens, or forward passes.
Does ActObs improve results immediately after SFT?
No. The paper shows the SFT variants perform nearly identically on Terminal-Bench 2.0 right after fine-tuning (e.g., 4.5 vs 4.4 pass@1 on Qwen3-4B). The differences appear only after applying the identical GRPO reinforcement-learning procedure on top of each initialization.
Why train a model on tokens it will never generate at deployment?
The authors argue that predicting an observation forces the model to represent what the preceding action does to the environment, so each trajectory serves as both a policy example and a transition example. Action-only SFT is shown to erode this consequence-prediction ability below even the base model, leaving a policy that explores more poorly under reinforcement learning.
What are the main caveats about the evidence?
Each configuration was trained once, and the reported bootstrap errors cover only evaluation sampling noise, not training-run variance; some reported gaps (e.g., 4B pass@16 +1.1) are within overlapping standard errors. Experiments cover only Qwen3 models, one synthetic teacher, and terminal/code-editing tasks, and code is promised only upon acceptance.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure curves (Figs. 2-9) are only available via captions and in-text numbers; visual trajectory shapes could not be independently verified from the extraction.
- Tables were embedded as linear text; numbers were cross-checked between abstract, body, and appendices and are consistent, but minor transcription risk remains.
- The extracted text includes the full paper through appendices (A-I) and references; no sections appear to be missing.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.