Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack
Sohir Maskey, Philipp Scholl, Jonas Knupp, Pit Neitemeier, Sascha Wirges
cs.AI · cs.CL
Quick verdict
Shows better-pretraining checkpoints can rank worse after SFT in a 30B MoE pipeline; credible in direction, though single-seed with a thin density link.
Summary
Does the checkpoint with the best pretraining metrics remain the best starting point for later training stages? Using a 30B-parameter mixture-of-experts pipeline (7.5T-token pretraining followed by mid-training, long-context adaptation, and SFT), the authors compare three source checkpoints: constant learning-rate pretraining (CONSTANT), a variant with the learning rate decayed over the final 800B tokens (COOLDOWN), and a weighted average of CONSTANT checkpoints (MERGE). COOLDOWN improves every pretraining signal — validation loss 1.648 versus 1.734 — yet after an identical downstream recipe its post-SFT aggregate trails CONSTANT, 0.247 versus 0.360. Across eleven trajectories, mid-training aggregates correlate only weakly with post-SFT outcomes (r = 0.473, not significant), while long-context aggregates correlate strongly (r = 0.884). The authors link the reversal to solution density, the fraction of Gaussian weight perturbations that retain benchmark performance: at sigma = 0.005 and threshold 0.90 on GSM8K, CONSTANT, MERGE, and COOLDOWN retain 27%, 13%, and 0%. Audits show COOLDOWN's gap is amplified by a stopping failure — repetitive generations hit the 32k-token cap — while its HumanEval+ score partly reflects last-block extraction; excluding HumanEval+ the gap shrinks to 0.016. Limitations: a single model family, one seed per setup, solution density measured on two benchmarks at three checkpoints, and no causal claim.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A pretraining checkpoint (COOLDOWN) that dominates every available pretraining signal produces a substantially worse post-SFT model (aggregate 0.247 vs 0.360) under an identical downstream recipe.
- Intermediate aggregate rankings are unstable after mid-training (r = 0.473, not significant) but largely stabilize after long-context adaptation (r = 0.884) across eleven trajectories.
- Source checkpoints with higher solution density — performance retained under Gaussian weight perturbations — are the ones that perform better after the downstream stack.
- COOLDOWN's largest deficit reflects a repetitive-generation stopping failure rather than absent latent capability: first-block extraction raises HumanEval+ from roughly 5-7% to roughly 60%.
Strengths
Production-scale evidence with rare transparency
A complete 30B MoE stack (7.5T-token pretraining, 100B mid-training at 8k, 100B long-context at 64k, 10B SFT) with architecture (Table 3), AdamW/Muon optimizer split (Appendix B.3), and exact token budgets disclosed — evidence of this kind is almost never published.
Exemplary failure forensics (Sections E-F)
zlib-to-raw compression ratios (0.019-0.044 for COOLDOWN vs 0.288-0.328 for CONSTANT on IFBench/AIME), first- vs last-block HumanEval+ re-execution in a network-disabled container reproducing every original verdict (Table 10), compact-answer rescoring (Table 9), and qualitative repetition examples with exact repeat counts.
Candid confound handling
Careful correlation analysis
Conceptual clarity on the metric itself
Weaknesses
Single-seed headline with no variance estimate
Solution-density link rests on three checkpoints
Low reproducibility
Stopping pathology not localized to a stage
Perturbation parameterization unspecified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether checkpoint selection by current pretraining metrics predicts which checkpoint yields the best final model after the remaining training stack. Three sources come from one 30B-parameter MoE run: CONSTANT (constant learning rate), COOLDOWN (LR decayed to 10% over the final 800B of 7.5T tokens; both share the first 6.7T), and MERGE (a linearly weighted average of 20 CONSTANT checkpoints, Appendix B.5). Claims: (1) conventional metrics misrank checkpoints — Table 1 shows COOLDOWN wins train loss (1.627 vs 1.717), validation loss (1.648 vs 1.734), and the pretraining aggregate (0.440 vs 0.415), yet after an identical mid-training, long-context, and SFT pipeline its post-SFT aggregate is 0.247 against CONSTANT's 0.360; (2) intermediate rankings are unstable after mid-training (r = 0.473, not significant) but largely stabilize after long-context adaptation (r = 0.884) across eleven trajectories; (3) solution density — the fraction of Gaussian weight perturbations retaining benchmark performance — separates the more trainable sources; (4) COOLDOWN's largest deficit is a stopping/repetition failure rather than missing capability.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of the paper Good Pretraining, Bad SFT?
The paper shows that in a 30B-parameter mixture-of-experts training pipeline, a pretraining checkpoint chosen by better loss and benchmark scores (the learning-rate-decay COOLDOWN variant) produced a clearly worse model after the standard mid-training, long-context, and SFT stages than a constant-learning-rate checkpoint (post-SFT aggregate 0.247 vs 0.360). Its point is that checkpoint scores measure current quality, not how well a checkpoint survives later training.
What is solution density in this paper?
Following Gan and Isola (2026), solution density is the fraction of small Gaussian perturbations of a model's weights that keep a benchmark score above a threshold (for example 90%) of the unperturbed score. In the paper, the checkpoints that survived downstream training better (CONSTANT and MERGE) retained GSM8K performance under perturbation far more often (27% and 13% at noise scale 0.005, threshold 0.90) than COOLDOWN (0%).
Why did COOLDOWN fail on HumanEval+?
Its generations became highly repetitive, emitting thousands of nearly identical code blocks until hitting a 32,768-token cap; the official evaluator extracts the last code block, which was usually a failing repeat. Extracting the first block instead raised pass@1 from about 5-7% to about 60%, showing the code ability existed but the model could not stop generating.
Can this paper's results be reproduced?
Not directly. The study relies on a proprietary 30B MoE pretraining run of 7.5 trillion tokens on 512 GPUs with a private data mixture, and no code or checkpoints are stated as released. Independent replication would require comparable industrial-scale compute; only the Gaussian perturbation evaluation procedure itself is simple enough to reimplement.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 appear only as captions in the extracted text; quantitative curve readings were taken from Tables 11-12 rather than from the plotted data.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.