Rufus-Air: An Open LLM Post-Training Recipe
Chia-Yuan Chang, Renyuan Cheng, Rui Feng, Xiaotian Han, Yuan He, Hongye Jin, Linwei Li, Shiyang Li, et al.
cs.CL · cs.AI · cs.LG
Quick verdict
Open 8-stage post-training recipe on GLM-4.5-Air-Base beats the official GLM-4.5-Air release; unusually candid, but stage-order rationale is untested.
Summary
Rufus-Air is a fully documented post-training recipe applied to the public GLM-4.5-Air-Base MoE checkpoint (106B total, 12B active), organized as a serial eight-stage pipeline: SFT, Reasoning RL, Coding RL, Instruction-Following RL, General Agent, Coding Agent, Search Agent, and RLHF. The authors disclose the data (17 public SFT datasets, 9M samples, 27B training tokens), reward designs, difficulty filtering, and infrastructure (Slime/Megatron/SGLang stack, Firecracker sandboxes, token-in/token-out rollouts), using no new human annotation. The final model beats the official GLM-4.5-Air release on every reported benchmark except Arena-Hard v2 Creative Writing — most visibly on instruction following (IFBench 76.9 vs 33.6) and agentic tasks (Tau2-Telecom 93.0 vs 32.7) — and is competitive with similarly sized open models such as INTELLECT-3 and Nemotron-3-Super. Main limitations: the stage-ordering principle (reward reliability) comes from training experience and is not ablated; several benchmarks (Terminal-Bench, SWE-bench, Seal-0) are measured only at their own stage and at the final checkpoint, so attribution is incomplete — the Coding Agent's SWE-bench gain does not even survive to the shipped model (67.8 → 65.6); AIME drops after Reasoning RL are dismissed as noise via the authors' own 4.6-point re-run variance; and no confidence intervals or significance tests accompany any comparison.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 0 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A diverse, high-quality SFT stage alone already beats the official GLM-4.5-Air release on IFEval, IFBench, and both AIME years, establishing the capability floor that RL refines (Table 4).
- Difficulty filtering that keeps prompts in a productive band (drop >80% pass rate, drop or retain never-solved per stage) is the most important choice in every outcome-reward RL stage.
- Ordering stages by reward reliability - hard verifiable rewards first, judge- and reward-model-based last - limits exposure to reward hacking (§2.1).
- Infrastructure choices (token-in/token-out rollouts, Firecracker sandbox snapshots, Rollout Routing Replay, large batches) are load-bearing parts of the recipe, not implementation details (§4).
- The finished Rufus-Air improves over the official GLM-4.5-Air release on every reported benchmark except Arena-Hard v2 Creative Writing and is competitive with similarly sized open models (Table 1).
Strengths
Exceptional disclosure granularity
Stagewise before/after tables (Tables 2, 4, 6, 8-12, 14), commit-pinned stack (Table 15), per-stage hyperparameters (Table 16), and honest operating costs (~$10K/month sandbox fleet, per-pass search API pricing in §6) — details almost never disclosed in this genre.
Candor about negative results
The Coding Agent's SWE-bench gain (65.60→67.80) is shown not surviving to the shipped model (65.6, §3.6); three of four RLHF prompt datasets are reported as failures (§3.8); AIME drops after Reasoning RL are printed rather than hidden (Table 6).
Rigorous, quantified contamination controls
Principled, uniform learnability filtering
Evaluation-sensitivity made explicit
Weaknesses
Central stage-order claim is unablated
Incomplete attribution across stages
No statistical rigor
Acknowledged protocol inconsistencies
Release scope of 'open' recipe is unclear
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Rufus-Air (arXiv 2609.29421) is Amazon's end-to-end post-training recipe on the public GLM-4.5-Air-Base checkpoint (106B total / 12B active MoE), presented deliberately as sitting "between a research paper and an engineering experience report" (§1). Eight serial stages — SFT (9.01M samples, 27B loss-bearing tokens from 17 public datasets), Reasoning RL, Coding RL, Instruction-Following RL, General Agent, Coding Agent, Search Agent, RLHF — each initialize from the previous checkpoint. The paper advances four claims: SFT is a capability floor, not a warm-up; difficulty filtering keeps RL prompts productive; reward reliability dictates stage order; infrastructure is part of the recipe. The headline outcome (Table 1): Rufus-Air beats the official GLM-4.5-Air release on every reported benchmark except Arena-Hard v2 Creative Writing (53.0 vs 60.3), with dramatic gaps on IFBench (76.9 vs 33.6), Multi-challenge (65.8 vs 36.0), and Tau2-Telecom (93.0 vs 32.7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 0 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Rufus-Air?
It is an Amazon-published post-training recipe applied to the public GLM-4.5-Air-Base model (106B total / 12B active parameters). Eight stages run in sequence: SFT, Reasoning RL, Coding RL, Instruction-Following RL, General Agent, Coding Agent, Search Agent, and RLHF, using only public data and open-source components.
How does Rufus-Air compare to GLM-4.5-Air and other open models?
Per the paper's Table 1, it beats the official GLM-4.5-Air release on every reported benchmark except Arena-Hard v2 Creative Writing (53.0 vs 60.3). Biggest leads: IFBench 76.9 vs 33.6, Multi-challenge 65.8 vs 36.0, Tau2-Telecom 93.0 vs 32.7. Against Nemotron-3-Super it trails on Tau2-Airline, Creative Writing, and ties Terminal-Bench 2.1.
Is Rufus-Air's code and data available?
The recipe builds on open-source components (Slime, Megatron-LM, SGLang, Strands Agents) and 17 public datasets, and hyperparameters are published. However, the paper does not clearly state whether the Rufus-Air weights or its Rufus-Slime/Rufus-Gym infrastructure adaptations are released, and Search Agent training used an internal search backend.
How much compute does the Rufus-Air recipe need?
RL stages ran on 8–32 nodes of 8xH200 GPUs (64–256 GPUs). The paper also reports non-GPU costs: a sandbox fleet sized for ~10,000 concurrent sandboxes at roughly $10K/month, and a single BrowseComp evaluation pass costing a few hundred dollars in search API fees.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Four text spans were omitted in extraction (tail of the SFT decontamination chain, General Agent data intro, RLHF reward/training details, part of Appendix C/Table 17); judgments touching those are provisional.
- Figures are available only as captions; training-curve claims (reward dynamics, eval panels) cannot be independently verified from text.
- Six of ten Table 1 baseline columns are developer-reported (Appendix D); cross-model statements inherit those comparability limits.
- Report uses authors' own harness re-implementations for agentic benchmarks; exact reference-implementation numbers may differ.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.