Nonsmooth Optimization via Orthogonalized Momentum
Lexiao Lai, Tianyi Lin, Jiayu Zhang
math.OC · cs.LG
Quick verdict
Sharp result: fixed-β Muon can fail on convex Lipschitz aims (any β∈[0,1)); adaptive momentum/MAGD restores min{m,n}ε⁻²-optimal bounds; empirics near-parity.
Summary
This paper analyzes Muon-style orthogonalized momentum for locally Lipschitz, possibly nonsmooth matrix optimization using conservative fields compatible with backpropagation. Its first contribution is negative: for every fixed momentum factor β∈[0,1), the authors construct a convex Lipschitz objective on R^{m×n} (m,n≥2) for which Muon fails to approach the global optimum from almost every initialization, even when step sizes adapt to the full subgradient history; the failure can occur even along bounded iterates. This strictly extends Parshakova et al. [2026], whose adaptive-step result covered only β∈[0,1/2). The positive results show the obstruction is fixed momentum, not orthogonalization: coupling βt=1−τηt to a vanishing step size restores asymptotic convergence of inexact Muon for nonconvex objectives under a weak-Sard regularity assumption, via the Lyapunov function f(W)+τ⁻¹‖M‖nuc. The proposed MAGD mixes an inexact Muon branch with a gradient branch, weighting them by relative linearized progress; it retains nonconvex asymptotic convergence and achieves O(min{m,n}ε⁻²) oracle complexity in the convex setting, matched by a string-guessing-based lower bound, establishing optimal dimension dependence in operator-norm geometry. Experiments on synthetic spectra (condition number 10⁴), CIFAR-10 CNN training, and 286M/537M-parameter LLM pretraining show MAGD competitive with Muon; reported gains are small (e.g., 0.4291 vs 0.4294 validation loss in Table 3), and the analyzed schedules (βt→1, δt→0) differ from those benchmarked.
Review complete
Assessment at a glance
3
risk flags identified
● high 0
● medium 2
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- For every fixed momentum factor β∈[0,1) and any m,n≥2, there is a convex Lipschitz objective on R^{m×n} where Muon fails to approach the global optimum from almost every initialization, even with step sizes adapting to the full subgradient history (Theorem 3.1).
- Coupling momentum to step size (βt=1−τηt, ηt→0, Σηt=∞) makes inexact Muon converge asymptotically for nonconvex locally Lipschitz objectives under weak-Sard regularity, with Mt→0 (Theorem 3.2).
- MAGD retains asymptotic convergence in nonconvex settings and achieves O(min{m,n}ε⁻²) oracle complexity for convex Lipschitz objectives, matched by an Ω(min{m,n}(GR/ε)²) lower bound in operator-norm geometry (Theorem 3.6).
- Experiments on synthetic spectra, CIFAR-10, and 286M/537M-parameter LLM pretraining show MAGD is a practical, competitive alternative to Muon.
Strengths
Sharp negative result closing an open gap
Theorem 3.1 proves almost-everywhere failure for every fixed β∈[0,1) with arbitrary history-adaptive step sizes, where Parshakova et al. [2026] covered only β∈[0,1/2); the proof is constructive, with an explicit diagonal piecewise-linear objective and a delicate Diophantine-style separation argument (choosing μ outside a countable set so g(μ)>0).
Correct diagnosis with an elegant Lyapunov analysis
The coupled schedule βt=1−τηt yields φ(W,M)=f(W)+τ⁻¹‖M‖nuc with dφ/ds=−‖M‖nuc along the differential inclusion (Theorem 3.2 proof), and the inexact-polar handling via singular-value truncation (constructing P̃t+1∈∂‖·‖nuc(M̃t+1)) is technically careful.
Matching upper and lower complexity bounds
Multi-scale empirical validation
Weaknesses
Analyzed algorithms differ from benchmarked algorithms
Marginal, untested empirical gains over Muon
Asymptotic-only nonconvex guarantees
Added hyperparameters without sensitivity analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether Muon's orthogonalized-momentum update survives the loss of smoothness, working on R^{m×n} with locally Lipschitz objectives modeled by conservative fields (Assumption 2.2 adds convex-valuedness and the weak Sard condition). Three claims: (1) Negative — Theorem 3.1: for every fixed β∈[0,1), any m,n≥2, there is a convex Lipschitz f (built from |[W]11+[W]22| and hinge terms of the diagonal gap) such that Muon with δt≡0 and *arbitrary* step sizes adapting to the full subgradient history satisfies inf_t f(Wt) > f⋆ for almost every W0; the abstract adds that failure can occur along bounded iterates. This strictly extends Parshakova et al. [2026], whose adaptive-step result covered only β∈[0,1/2). (2) Repair — Theorem 3.2: under the coupling βt=1−τηt (Eq. 1.1) with ηt→0, Σηt=∞ and vanishing polar error, inexact Muon's cluster points are Df-stationary, Mt→0, and objective values converge; under convexity (Assumption 2.3) they reach the global optimum. (3) Method plus complexity — MAGD (Algorithm 1) mixes a Muon branch Ut and a gradient branch St with a sigmoid weight driven by relative linearized progress; Theorem 3.4 gives asymptotic convergence in two regimes, and Theorem 3.6 establishes Θ(min{m,n}(GR/ε)²) matching bounds in operator-norm geometry via a string-guessing reduction (Proposition 3.9).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 3 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MAGD?
Muon Advised by Gradient Descent: an optimizer that runs a Muon (orthogonalized momentum) branch and a plain gradient descent branch, then mixes them with a weight chosen by an online-experts-style rule comparing their predicted linearized decreases, so plain gradient steps safeguard the orthogonalized Muon update.
Why does Muon fail on nonsmooth objectives according to this paper?
The paper argues the failure comes from combining fixed momentum with orthogonalization: at a kink the gradient can change abruptly, a fixed β propagates that change at a rate independent of the step size, and orthogonalization turns any nonzero momentum into a full-size normalized step, so iterates can stay away from the optimum on a convex Lipschitz objective.
Does MAGD beat Muon?
In the reported experiments it is essentially competitive rather than clearly better: CIFAR-10 validation loss 0.4291 vs 0.4294 (Table 3, within about one standard error over 100 seeds), and on the synthetic benchmark it beat the better of Muon and gradient descent on five of seven spectrum families.
What is the optimal oracle complexity for convex nonsmooth optimization in operator-norm geometry?
The paper establishes Θ(min{m,n}(GR/ε)²) oracle calls (for the regime (GR/ε)²>1), with MAGD-style upper bounds matched by a string-guessing-based lower bound, showing the min{m,n} dimension dependence is optimal (Theorem 3.6).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with explicit '[...sampling gap...]' markers: statements of Theorems 3.2/3.5 and Lemma 3.3, Figures 1–3, Tables 1–2, and part of the references were not in the shown span, so judgments on those items rely on in-text cross-references only.
- No code-availability statement appears in the provided text; one may exist in a section inside a sampling gap of the full paper.
- Figure 2 and Figure 3 content (synthetic curves, LLM loss curves) is not visible in the excerpt, so quantitative LLM conclusions are limited to the prose description.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.