T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks
Junyao Yang, Yucheng Shi, Zhongzhi Li, Ruhan Wang, Zongxia Li, Haitao Mi, Leowei Liang
cs.LG · cs.AI
Quick verdict
Claims TITO+R3 and dense verifier rewards lift a 122B MoE to 64.0% on Terminal-Bench 2.1; recipe credible, but no CIs and test-set checkpoint selection.
Summary
The paper introduces T1, a Mixture-of-Experts model with 122B total and 10B active parameters, post-trained from Qwen3.5-122B-A10B using PPO to operate a real shell in cloud sandboxes on tasks of up to 300+ tool-call turns, rewarded by executing each task's own per-assertion verifier. The recipe combines three elements: (1) an aggressively warm-started critic trained at 30× the actor learning rate, with a dense process reward counting passing assertions on a fixed global scale; (2) MoE stabilization through TITO, which trains on the exact sampled token identifiers with turn-boundary drift repair, and R3, which records and replays the sampler's per-token expert routing across all 48 layers, cutting the training-to-inference log-probability gap from 0.021 to 0.013; (3) a synthesized corpus (T1-15k, drawn from RST-38k) claimed fully out-of-distribution relative to Terminal-Bench 2.1. Three campaigns are documented: a binary-reward run from the base model that never beat its SFT checkpoint (43.8%→47.2% vs 49.4%); dense reward on unfiltered RST-38k (59.9%); and the production run on T1-15k reaching 64.0% resolved at step 110, versus GPT-5.4 at 54.8% under the same Terminus-2 harness. T1 scores 27.9% on Long-Horizon Terminal-Bench and 38.0% on Terminal-Bench Hard. Visible limitations: no confidence intervals or seed variance despite the 89-task benchmark, best-of-repeated-evaluations checkpoint selection on the test set, and selective baseline emphasis in headline claims.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PPO on executed per-assertion verifier outcomes raises Qwen3.5-122B-A10B from 49.4% (SFT checkpoint) to 64.0% resolved on Terminal-Bench 2.1, from a 43.8% base model.
- TITO plus R3 reduce the training-to-inference log-probability gap from 0.021 to 0.013 with zero token drift in the loss region, stabilizing sparse MoE agentic RL.
- A dense process reward counting passing assertions on a fixed global scale, with a critic warm-started at 30× the actor learning rate, is necessary: the binary-reward campaign never exceeded the SFT baseline.
- The T1-15k training corpus is fully out-of-distribution with respect to Terminal-Bench 2.1, so gains reflect capability transfer rather than benchmark fitting.
- Under the same Terminus-2 harness, T1 surpasses GPT-5.4, DeepSeek-V4-Flash and Claude Opus 4.6 and approaches Claude Opus 4.7 on Terminal-Bench 2.1, and reaches 27.9% on Long-Horizon Terminal-Bench and 38.0% on Terminal-Bench Hard.
Strengths
Candid negative results and failure attribution
The first binary-reward campaign from the base model never beat its SFT checkpoint (43.8%→47.2% at iteration 30, below 49.4%), and Section 9 documents instructive failures: critic explained variance improving without policy gains (§9.1) and a context-compaction bug failing 98.3% of full-summary attempts, raising average turns 22→30 (§9.2). Two failed reward-shaping variants are reported alongside the working one.
Formally grounded MoE stabilization stack
Section 4.1 separates token fidelity (Eq. 3) from routing fidelity (Eq. 4) for the 8-of-256 expert router, and Appendix B (Algorithm 4) turns stitching into asserted invariants — per-turn offsets, gap repair confined to masked positions, identical sharding, cursor discipline for recomputation — rather than assumptions.
Executable reward with anti-hacking hygiene
Careful same-harness evaluation discipline
Internally consistent results chain and clean case study
Weaknesses
No uncertainty quantification on n=89 benchmark
Checkpoint selection on the test benchmark
Selective LHTB baseline emphasis
Harness-training asymmetry underweighted in headlines
Compute and release scope not stated in visible text
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is T1 and how is it different from a fine-tuned coding model?
T1 is a 122B-total/10B-active Mixture-of-Experts model (from Qwen3.5-122B-A10B) post-trained purely with PPO to operate a real Linux shell in cloud sandboxes, with up to 300+ tool-call turns per task. Unlike preference-tuned models, its reward comes from executing each task's own verifier, assertion by assertion.
How does T1 compare with frontier models on Terminal-Bench 2.1?
The paper reports 64.0% resolved on Terminal-Bench 2.1 under the Terminus-2 harness, versus GPT-5.4 at 54.8%, DeepSeek-V4-Flash at 56.9%, Claude Opus 4.6 at 63.8% and Claude Opus 4.7 at 66.1%. The authors note T1 was RL-trained for exactly this harness while the frontier models were not.
What are TITO and R3 in the T1 paper?
TITO makes the trainer score exactly the token identifiers the inference engine sampled, repairing drift at turn boundaries; R3 records the sampler's per-token expert choices at every MoE layer and replays them during training. Together they cut the training-to-inference log-probability gap from 0.021 to 0.013 with zero token drift in the loss region.
Is T1's training data contaminated by the evaluation benchmark?
The authors state the synthesized corpus (T1-15k, built via RST from isolated seeds) is disjoint from Terminal-Bench 2.1, and gains therefore reflect transfer rather than benchmark fitting; the detailed disjointness audit procedure is not fully shown in the publicly sampled text.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Large parts of the body fall inside marked [...sampling gap...] regions (parts of §3.2, §4.4–4.5, §5.1–5.4, §6.3, Case Study I, §7, §9.3–9.4, §10, and parts of Table 1/appendices); judgments avoid relying on those sections.
- Table 1 is cut off mid-list by a sampling gap; the LHTB column values for T1 and GLM-5.1 are not visible in the provided text (T1's 27.9 appears in abstract/intro/§2.4 prose).
- No explicit weights/code/data release statement appears in the provided text; only 'Project Page' and 'HuggingFace' link labels are visible near Figure 1, so release scope cannot be confirmed from the excerpt.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.