Train What You Deploy:Token-Faithful Post-Training of a Production Coding
Cheng Li, Jiexiong Liu, Yixuan Chen, Chi Hong
cs.AI
Quick verdict
Sound certification theory (C-DPPO) atop a token-faithful agent-training coupling, but the headline +3.0 gain rests on single runs below the noise floor.
Summary
The paper tackles a fidelity problem in RL post-training of LLM agents: training trajectories must come from the same production scaffold used at deployment. It couples KCoder, a proprietary Rust terminal-coding agent, with the slime Megatron-SGLang framework through a 'negotiated coupling': the trainer owns sampling and re-renders every prompt to exact token ids; the agent runs a hardened training mode machine-checked to emit zero background model calls; and a message-tree linearizer classifies re-tokenization drift (Clean/Realign/Fork) and assigns loss only to provably sampled spans. On this observation interface the authors build C-DPPO, extending DPPO with a tight two-sided total-variation certificate for Binary/Top-K coarsenings, an adaptive-K termination rule, a sequence-level divergence budget with adaptive per-position thresholds, and a mask robust to bounded log-probability error; they also prove tail mass alone cannot certify KL. In a matched study (Baize5B and Baize10B, eight H20 GPUs, 350 steps, TMax-15K training, frozen TMax-100 evaluation), C-DPPO scores 37 vs 34 and 44 vs 41 — a +3.0-point difference at both scales — with 99.4%/99.1% complete audit records. The authors candidly note each delta sits below the roughly ±4.8-point binomial noise floor, rests on a single run per pair, and carries seed and mask-selection confounds; the result is descriptive, not significant. No code release is stated, limiting reproduction.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A negotiated coupling — trainer-owned sampling on exact token ids, a machine-checked zero-background-request training mode, and fail-closed provenance headers — makes RL post-training of a production agent token- and control-faithful by construction (§4.1–4.2).
- C-DPPO supplies a tight two-sided TV certificate for Binary/Top-K coarsenings with adaptive-K termination, and tail mass alone cannot certify KL (Theorem 5.1, Eqs. 4–5).
- Per-token certificates compose into a sequence-level divergence budget with adaptive δt, yielding a policy-improvement bound J(π) − J(μ) ≥ L′μ(π) − 4ξB (Theorem 5.3, Eq. 9).
- On matched Baize5B/10B pairs (350 steps, frozen TMax-100), C-DPPO yields a direction-consistent +3.0-point gain over Standard DPPO (37 vs 34; 44 vs 41), which the authors explicitly frame as descriptive, below the ~±4.8-point binomial noise floor (§6.3, §7).
Strengths
Correct, tight certification theory with an honest KL boundary
Theorem 5.1 gives the exact coarsening gap DTV = D̂S + min{PS,NS} with a tight observable envelope τS = min{μ(̄S), π(̄S)}; the Appendix A proof is three lines and sound. The KL chain-rule remainder (Eq. 4) plus the two-token-tail impossibility construction (remainder m·log(1/ε) diverging as ε→0) is a clean boundary result rather than an overreach.
The fidelity contract is machine-checked, not documented
§4.1: a regression test asserts exactly two provider requests across a tool round-trip in training mode; missing depth/replay headers fail closed with HTTP 400; the binary is SHA-256-pinned per template build; the training flag cannot be re-enabled via config files. This converts silent scaffold drift into loud rejection.
Conservative-by-construction mask semantics
Reward hygiene above community norm
Candid statistical self-assessment and an audit that separates mechanism from outcome
Weaknesses
Empirical evidence is one matched pair per scale with a sub-noise effect
Treatment confounds the update rule with training-data selection
No external RL baselines
Reproducibility is structurally limited
Abstract slightly overstates the audit
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper makes two coupled contributions. First, a systems contract: a 'negotiated coupling' between the proprietary KCoder terminal-coding agent (Rust) and the slime Megatron-SGLang post-training stack, in which the trainer owns sampling on exact token ids (§4.1: input_ids with return_logprob, session-id-as-API-key consistent-hash routing, a hardened training mode whose zero-background-request property is machine-checked via an exactly-two-requests-per-tool-round-trip regression test, and depth/replay negotiation headers that fail closed with HTTP 400). Second, C-DPPO: a certification layer over DPPO [21] comprising an exact Binary/Top-K TV coarsening gap with a tight observable envelope (Theorem 5.1), adaptive-K termination (Corollary 5.2), a sequence-level divergence budget with adaptive δt (Theorem 5.3), and a mask robust to bounded log-probability error ε=0.05 (Theorem 5.4), plus a KL impossibility result. The empirical claim is deliberately narrow: one matched Standard-DPPO/C-DPPO pair per scale (Baize5B, Baize10B), 350 steps, frozen TMax-100 evaluation, +3.0 points at both scales (37 vs 34; 44 vs 41, Table 2), explicitly framed as descriptive.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is C-DPPO?
C-DPPO (Certified Divergence Proximal Policy Optimization) is an extension of DPPO, a trust-region method for LLM reinforcement learning. It adds a computable two-sided bound on how far the cheap Binary/Top-K divergence approximation is from the true total-variation divergence, an adaptive rule for how many top tokens to track before deciding inside/outside the trust region, a divergence budget spread across a long agent trajectory, and a token mask that remains safe when the log-probabilities themselves carry small declared errors.
How much better is C-DPPO than standard DPPO?
In the paper's matched study, C-DPPO scored 37/100 vs 34/100 on the TMax-100 benchmark with 5B models and 44/100 vs 41/100 with 10B models — a +3.0-point gain at both scales. The authors stress this is below the ~±4.8-point binomial noise floor of a 100-task evaluation and comes from a single run per setting, so it is a direction-consistent observation rather than a statistically significant result.
Is the code or the KCoder agent publicly available?
No release is stated in the paper. The stack builds on open components — the slime post-training framework, SGLang, Megatron-LM, and the public TMax-15K task corpus — but KCoder itself is described as a proprietary production binary, and no repository, run manifest, or audit records are linked, so full replication is currently not possible from public artifacts.
Why train the production agent instead of a simplified loop?
Simplified research loops optimize the policy under a different scaffold than the one deployed: production harnesses rewrite history via context compaction, inject memory and skill blocks, re-serialize tool schemas, and issue background model calls. Tokens learned under a minimal loop are conditioned on the wrong distribution, and background calls can be mistaken for the policy's voice. This paper instead makes the deployed binary itself generate training data, with the trainer owning exact token sampling and a machine-checked guarantee that no background calls contaminate the training signal.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3–6 appear in the extraction only as captions and axis ranges; plotted curve values cannot be verified beyond the in-text descriptions (e.g., 'settling near 0.48').
- PDF extraction slightly mangles some equation glyphs (e.g., the coarsened-divergence symbol), but all theorem statements, proofs, and numbers remain recoverable; the text ends naturally at reference [23] with no truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.