Optimizers for Diffusion Models: A Controlled Benchmark
Arman Bolatov, Egor Shulgin, David Li, Abduragim Shtanchaev, Sebastian U. Stich, Maxim Panov, Eric Moulines, Peter Richtárik, et al.
cs.LG
Quick verdict
First controlled optimizer benchmark for discrete diffusion; well-run and reproducible, but text-task budgets are just 3% of published training length.
Summary
This paper presents, to the authors' knowledge, the first controlled optimizer benchmark for discrete diffusion models. Seven optimizers (AdamW, Lion, Muon, SOAP, MARS, MARS-M and Schedule-Free AdamW) are compared across four diffusion formulations, each on a task with published reference values: masked diffusion on text8, uniform diffusion (UDLM) on QM9, uniform diffusion via the Gaussian duality (DUO) on LM1B, and Gaussian DDPM on CelebA-64. Every optimizer shares one tuning protocol (Optuna TPE with Hyperband pruning; 100 or 50 trials at 20% of budget), and winning configurations are retrained at full budget over three seeds, roughly 1400 GPU-hours in total. Tuned AdamW is never beaten by a resolved margin on text8 or CelebA-64; on QM9, SOAP, Lion and Muon win by resolved but small margins (perplexity 2.066-2.069 versus 2.072, 0.1-0.3%); on LM1B, Muon (61.6) and MARS-M (62.2) finish six to seven perplexity points ahead of AdamW (68.2). Rankings do not transfer across formulations (Kendall tau from -0.14 to +0.71), Schedule-Free finishes last on all four tasks, Lion diverges on LM1B, and sample-quality metrics (FID, FCD) disagree with loss orderings. Main limitations: text8 and LM1B train for only about 3% of published budgets, so those orderings are provisional; one model size and batch size per task; three seeds with some marginal resolutions.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- This is, to the authors' knowledge, the first controlled optimizer benchmark for discrete diffusion models, run alongside Gaussian image diffusion.
- A carefully tuned AdamW is a strong default but is beaten by resolved margins on two of four tasks: QM9 (SOAP, Lion, Muon by 0.1-0.3% perplexity) and LM1B (Muon and MARS-M by six to seven perplexity points).
- Optimizer rankings do not transfer across diffusion formulations (Kendall tau from -0.14 to +0.71; five of six task pairs at or below +0.43).
- Optimizers validated on autoregressive pretraining transfer at least partially: Muon, MARS-M and SOAP each beat tuned AdamW on at least one diffusion formulation.
- Matrix-preconditioning methods' cost appears in wall-clock time rather than FLOPs (SOAP adds 0.3% FLOPs but 190% wall-clock per step on the CelebA UNet).
Strengths
Genuine gap, cleanly scoped question
MDLM, UDLM and DUO all inherit AdamW without comparison (Section 1); the benchmark ranks optimizers strictly within each formulation rather than conflating optimizer and formulation effects, with Gaussian DDPM on CelebA-64 as a control (Section 4.2).
Above-average benchmarking discipline
Identical Optuna TPE + Hyperband search per optimizer (Section 4.3), winners retrained at full budget on three seeds, no reporting of best trajectory points, full search budgets and every selected hyperparameter published (Table 5), and all ten grid-edge selections enumerated (Appendix F).
Evaluates likelihood and sample quality, not just loss
Honest accounting of truncation and deviations
Practical cost analysis that FLOP counts hide
Weaknesses
Headline LM1B result rests on 3% of the published budget
Thin statistical margins behind the QM9 win claim
Grid-edge selections weaken the CelebA-64 conclusion
Implementation deviations never ablated
Sample-quality evidence is thin and one-sided
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper claims the first controlled optimizer benchmark for discrete diffusion models, run alongside Gaussian image diffusion. Seven optimizers — AdamW, Lion, Muon, SOAP, MARS, MARS-M and Schedule-Free AdamW — are compared across four formulations, each on a task with published reference values: masked diffusion (MDLM) on text8, uniform diffusion (UDLM) on QM9, uniform diffusion via the Gaussian duality (DUO) on LM1B, and Gaussian DDPM on CelebA-64 (Section 4.2). Three headline claims: tuned AdamW is a strong default but beaten by resolved margins on two of four tasks; the winner changes with the formulation; and optimizers validated on autoregressive pretraining transfer at least partially. The scope — rank optimizers within a formulation, never across — is explicit, and most claims are calibrated to what was measured, with the notable exception of the abstract stating the LM1B finding without the provisional qualifier Section 5.3 itself applies.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which optimizer should I use to train a diffusion model?
According to this benchmark, a carefully tuned AdamW remains a strong default: it is not beaten by a resolved margin on text8 or CelebA-64. However, Muon, MARS-M and SOAP each beat tuned AdamW on at least one formulation — the largest gains are on LM1B, where Muon reaches perplexity 61.6 versus AdamW's 68.2 — and rankings change across formulations, so the paper recommends testing alternatives on your specific task and judging by a sample metric as well as the loss.
Is this really the first optimizer benchmark for discrete diffusion?
The authors claim so with the caveat 'to our knowledge'. Prior optimizer benchmarks (Schmidt et al. 2021, AlgoPerf, Semenov et al. 2025) contain no diffusion tasks, and the closest related work (Schaipp 2025) benchmarks six optimizers on a Navier-Stokes denoising task, not discrete diffusion — though that paper anticipated the Schedule-Free loss-vs-samples effect measured here.
Did any optimizer actually beat AdamW in this study?
Yes, on two of four tasks. On QM9, SOAP (2.066), Lion (2.068) and Muon (2.069) beat tuned AdamW (2.072) by 0.1-0.3% perplexity. On LM1B, Muon (61.6), MARS-M (62.2) and MARS (66.6) beat AdamW (68.2). On text8 and CelebA-64, no alternative beat AdamW by a resolved margin, and Schedule-Free finished last on all four tasks.
Were the models trained to full convergence like the original diffusion papers?
No, except for QM9, which is a full-budget replication of UDLM (25k steps, batch 2048). text8 and LM1B train for 30k steps versus the roughly 1M steps of published results (about 3%), and CelebA-64 is one fifth of the DDIM paper's training length, so the text8 and LM1B orderings are explicitly provisional; the paper's text8 bits-per-character (about 1.81) is also far from published values (1.37-1.45).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as extracted captions and prose descriptions; curve-level claims (e.g., Schedule-Free's mid-training lead and the decay-phase crossover) cannot be verified from the plots themselves.
- The released code and run logs were not inspected; reproducibility judgments rest on the paper's description (Appendix K) and pinned versions.
- The metadata author list omits Martin Takac, who appears in the full text with affiliations — a minor metadata discrepancy.
- Table 1's shading/arrow significance markers depend on per-task spread values not fully recoverable from the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.