Temperon: Full-Time SAM Quality at a Third Less Wall-Clock
cs.LG · cs.CV
Quick verdict
A scheduled SGD→SAM+Muon hand-off matches full-time SAM at ~⅓ less wall-clock; refreshingly honest ablations, but n=5 tests and small scale cap generality.
Summary
The paper asks where an expensive training mode should be spent within a fixed epoch budget. Temperon runs plain Nesterov SGD as an 'explorer' for the first 43% of epochs, then makes one scheduled hand-off so a SAM-wrapped Muon refiner owns the entire final cosine anneal. On CIFAR-10/100, SVHN and Tiny ImageNet (WideResNet-110 at 4x width, 27.6M params, 5 seeds, timing reported as epochs-to-target x idle-GPU-calibrated per-epoch cost), it matches the best full-time-SAM recipe on final accuracy (Welch p>=0.28) while reaching the hardest common target 32-35% sooner on three of four datasets. Single-change ablations attribute the accuracy entirely to the Muon refiner (+0.85pp, p=0.005); the cyclic explorer shape and warm restarts are withdrawn as contributions. Re-running late-phase SAM at matched budget shows it is fastest to mid-level targets and wins Tiny ImageNet outright, defining the method's boundary. The allocation law transfers to a one-seed GPT-2 pretraining pilot (-29% wall-clock at matched loss) and GLUE fine-tuning (equivalence at ~1/3 the SAM cost, after a rho correction removes the apparent win except on MRPC). Main limitations: a single CNN architecture at 100 epochs, single-seed LM evidence, no ImageNet-scale test, and many uncorrected n=5 significance tests.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A single scheduled hand-off (plain-SGD explorer for the first 43% of epochs, then a SAM-wrapped Muon refiner owning the entire final cosine anneal) matches the best full-time-SAM recipe's accuracy on CIFAR-10/100, SVHN and Tiny ImageNet while reaching the hardest common target 32-35% sooner on three of the four.
- Exact attribution: the Muon refiner contributes the entire accuracy gain (+0.85pp, p=0.005); the allocation shape and cyclic restarts contribute nothing (the shape is 0.25pp worse than a plain switch) and are withdrawn as contributions.
- Late-phase SAM, re-run at matched budget, is fastest to every mid-level target and wins Tiny ImageNet outright; Temperon's win condition is a task where the Muon refiner buys a tier (0.83 CIFAR-100, 0.97 CIFAR-10) that no SGD-refined method reaches in any seed.
- The allocation law transfers: GPT-2 124M pretraining matches full-time SAM quality at -29% wall-clock (one seed), and RoBERTa GLUE fine-tuning is never worse than full-time SAM at roughly a third of its SAM cost, after a rho correction removes the apparent fine-tuning win except on MRPC.
- The hand-off cannot be timed adaptively from the trajectory under cosine schedules (plateau-then-surge), so the anneal-aligned scheduled switch is principled; economically, a Muon epoch costs a flat ~1.50x a SAM+SGD epoch across all four vision datasets.
Strengths
Self-critical attribution via single-change ablations
Section 5: Arm S (SGD refiner) lands at 0.8210+-0.0006 (-0.85pp vs Temperon, p=0.005) and is 0.25pp WORSE than the rival's plain mid-schedule switch (p=0.005); Arm T shows restarts are indistinguishable from a single cosine (p=0.611). The authors explicitly withdraw the allocation-shape and restart claims from earlier drafts — rare candor that makes the causal story (Muon refiner sets the accuracy tier) credible.
The strongest rival is measured, not strawmanned
Section 6 re-runs Zhou et al.'s late-phase SAM at matched SAM budget (57/100 epochs) in the same pipeline on all four datasets, reports it fastest to every mid target, and documents its outright win on Tiny ImageNet (0.7020 vs 0.7003, p=0.42; -13/-10/-7% to the 0.68/0.69/0.70 targets), stating the method's win condition as a boundary rather than hiding it.
Timing methodology confronts wall-clock noise head-on
Refuses a flattering GLUE headline after correction
Pre-registered negative result on adaptive switching
Weaknesses
Scale ceiling on every headline claim
Many uncorrected n=5 significance tests
The differentiating tier is reached by a minority of Temperon's own seeds
Boundary claim may hinge on an under-tuned baseline
Time-to-target metric is opaque for partially-reached targets
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Temperon and how does it work?
Temperon is a training recipe, not a new optimizer: it trains with plain Nesterov SGD for the first ~43% of the epoch budget, then makes one scheduled switch so that a SAM-wrapped Muon optimizer owns the entire final cosine-annealing phase. The idea is to pay SAM's ~2x per-step cost only late in training, where its generalization benefit concentrates, while a stronger refiner (Muon) handles the final anneal.
Does Temperon beat SAM?
It matches full-time SAM's accuracy on CIFAR-10/100, SVHN and Tiny ImageNet while reaching a hard accuracy target roughly a third sooner in wall-clock on three of the four datasets. It does not beat the best full-time SAM on accuracy, and on Tiny ImageNet a simpler late-phase-SAM baseline is faster and wins outright, so the recipe only pays off where the Muon refiner buys an accuracy tier cheaper methods cannot reach.
Why is it called Temperon?
The name comes from 'tempering' — the controlled heat treatment applied after quenching that sets a metal's final toughness. The analogy is that the expensive late phase governs what happens during the final learning-rate anneal, which sets the model's final quality.
Is code available for Temperon?
Yes. The paper states that a pip-installable implementation (pip install temperon), wrapping any pair of PyTorch optimizers, is released together with the full experimental record, including logs and the negative-control ablations the conclusions rest on.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only via captions and numbers quoted in the extracted text; curves, bands and error bars could not be independently inspected.
- The per-optimizer calibrated seconds/epoch constants are referenced but not tabulated in the provided text, so headline time deltas cannot be recomputed.
- Table 1's column alignment was partially flattened by extraction; the method-to-value mapping was reconstructed from surrounding prose and should be verified against the PDF.
- Review is based solely on the provided full text; no appendix or supplementary material was available.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.