Post-Training Science for Supervised Fine-Tuning
Charles O'Neill, Mudith Jayasekara, Harry Partridge
cs.LG · cs.CL
Quick verdict
970-run SFT sweep finds flat LoRA LR 1e-3, 98% of FullFT gains retained, ~2-epoch ceiling; rigorous but judge circularity limits generality.
Summary
The paper reports a large controlled empirical study of supervised fine-tuning (SFT) hyperparameters: roughly 970 eligible training runs that vary one lever at a time across nine dense models (Qwen3 0.6B–32B; Llama 3.2-1B/3B and 3.1-8B), three Qwen3 mixture-of-experts models up to 235B, and four anonymized customer SFT datasets, for both LoRA and full fine-tuning. Training data is generated by an iterative-SFT pipeline, so each task's judge is the criterion its examples were built to pass. Key findings: the optimal LoRA learning rate is flat at 1e-3 across 0.6–32B and both families, about 33× the full-fine-tuning optimum near 3e-5; the rule transfers across families and to a held-out 30B MoE, and holds blind on 80B and 235B. LoRA recovers a median 98% of full fine-tuning's improvement over the pretrained baseline; rank plateaus near 64 with α=32 best. Validation NLL ranks judged quality within a fixed recipe (standardized Spearman −0.38 to −0.88) but not across families, and Fisher-trace flatness adds nothing for in-domain selection yet tracks retained general capability. Loss follows a saturating power law in size, with MoEs entering at the geometric mean of active and total parameters. Past about two epochs, loss overfits and instruction-following erodes. Main limitations: four private datasets, judges not independent of the training data, a narrow full-fine-tuning grid, and some very small effect sizes.
Review complete
Assessment at a glance
0
risk flags identified
● high 0
● medium 0
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The optimal LoRA learning rate is flat at 1e-3 across 0.6–32B models in both Qwen3 and Llama families, roughly 33× the full-fine-tuning optimum near 3e-5, and the rule transfers unchanged to mixture-of-experts models up to 235B.
- LoRA recovers a median 98% of full fine-tuning's improvement over the pretrained baseline at 3.1–12.6% trainable parameters, though FullFT wins all 72 matched loss comparisons by small margins.
- Validation NLL reliably ranks judged downstream quality within a fixed (model, dataset, recipe) cell (standardized Spearman −0.38 to −0.88) but not across model families; Fisher-trace flatness adds nothing for in-domain selection yet tracks retained general capability (ρs = −0.71 on IFEval across optimisers).
- At fixed defaults, SFT loss follows a saturating power law in model size (LoRA exponent 0.31–0.40, FullFT 0.40–0.51), and all three MoEs enter the dense trend at roughly the geometric mean of their active and total parameters.
- Past about two epochs, validation loss overfits, judged task quality stops improving, and general instruction-following erodes; fresh examples beat repeated passes at a matched budget.
Strengths
Unmatched controlled scale with one-lever-at-a-time design
970 eligible training cells with fixed seeds, splits, and label masking, holding everything constant but the lever under study; the selection rule is validated on a held-out 30B MoE (13/16 cells pick 1e-3, mean regret 0.0011 nats vs the dense families' own 0.0026) and applied blind to 80B and 235B MoEs (§2.1, §2.7).
Explicit handling of the token-load confound and honest eligibility rules
The paper shows token load, not baseline difficulty, sets achievable loss (R² = 0.58/0.61 vs 0.01/0.02; §2.4) and responds by reporting improvement-over-baseline; the LR law is fit by the loss its predicted rates would incur rather than by noisy argmins (§2.2); the 38 excluded cells are fully accounted for (App. B).
Uncertainty discipline throughout
Falsifiable flatness test with a clean null result
Actionable, cost-quantified defaults
Weaknesses
Judge non-independence is structural, not incidental
Grid asymmetry between LoRA and FullFT arms
Narrow adapter study generalised to a default
Very small effect sizes drive recommendations without practical-significance analysis
Reproducibility is structurally limited
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a large empirical measurement paper on the practice of supervised fine-tuning (SFT), from a Baseten team. It runs a one-lever-at-a-time sweep — 970 of 1,008 planned cells eligible — over learning rate and batch size (§2), LoRA rank/alpha (§3), validation loss vs judged quality and Fisher-trace flatness (§4), scaling in size and data (§5), AdamW vs Muon (§6), and epochs (§7). The testbed: nine dense models (Qwen3 0.6–32B; Llama 3.2-1B/3B and 3.1-8B), three Qwen3 MoEs (30B-A3B, Next-80B-A3B, 235B-A22B), four anonymized customer datasets (security, leasing, support, docs), each with LoRA and full fine-tuning. Headline claims: the optimal LoRA LR is flat at 1e-3 across 0.6–32B and both families (~33× the FullFT optimum near 3e-5); LoRA recovers a median 98% of FullFT's improvement over baseline; loss ranks judged quality within a recipe but not across families; loss follows a saturating power law in size, with MoEs entering at the geometric mean of active and total parameters; and beyond ~2 epochs loss overfits while instruction-following erodes.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 0 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What learning rate should I use for LoRA fine-tuning?
The paper measures the optimal LoRA learning rate across Qwen3 (0.6B–32B) and Llama (1B–8B) models and four customer SFT datasets, and finds it essentially flat at 1e-3 everywhere — about 33× the full-fine-tuning optimum near 3e-5. Committing to 1e-3 without any tuning costs under 0.01 nats of validation loss versus per-cell tuning.
How many epochs should I fine-tune an LLM for?
Roughly two. Past that point validation loss overfits, the judged task score stops improving, and general instruction-following (IFEval) degrades — e.g. Qwen3-8B full fine-tuning on support fell from 0.73 at two epochs to 0.34 at eight. Adding fresh examples beats repeating the same data at a matched budget.
Does LoRA match full fine-tuning quality?
Not quite, but close. Full fine-tuning won all 72 matched comparisons on validation loss, yet by small margins: LoRA recovered a median 98% of FullFT's improvement over the pretrained baseline while training only 3.1–12.6% of parameters. The gap narrows further as models get larger.
What LoRA rank and alpha should I use?
The paper finds rank helps until about 64 and then plateaus (rank 128 gains only 0.0004–0.0008 nats), and α=32 was best in every tested cell at learning rate 1e-3 — so the default r=64, α=32 holds, with rank 32 as a cheaper option giving up at most ~0.003 nats. Caveat: rank was tested only on Qwen3-4B/8B on two datasets with the learning rate fixed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with marked gaps: parts of §2.5–§2.6, detail of §4.2, most §6 results and early §7 fall in sampling gaps; findings from those sections here rely on the visible abstract, introduction, conclusion, and appendix restatements.
- No code or data availability statement appears anywhere in the provided excerpt; one could exist within a sampling gap, so codeAvailable=false reflects the excerpt only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.