Muon Can Outperform Dedicated Continual Learning Methods
Sebastian George Sincari, Bogdan Alexandru Gheorghe, Antonio Barbalau
cs.LG · cs.CL
Quick verdict
IncLoRA+Muon matches O-LoRA/ELLA; Muon spreads updates over ~7 singular dirs vs AdamW's 1.4–1.8. Small-scale (T5) but honest and mechanistically convincing.
Summary
The paper asks whether the anti-forgetting constraint in LoRA-based continual learning must be task-aware, or whether a generic optimizer-level constraint suffices. The authors train plain incremental LoRA (IncLoRA) with Muon, an optimizer that orthogonalizes each update, and compare six method-optimizer combinations against O-LoRA and ELLA on the Standard CL Benchmark (T5-Large; three task orders, five seeds, 90 runs) and TRACE (Flan-T5-Large; one order, three seeds). IncLoRA+Muon reaches 81.24 OA versus 81.00 for O-LoRA+AdamW and 79.64 for ELLA+AdamW; on TRACE all Muon configurations beat all AdamW ones (31.36–31.80 vs 24.18–25.92 OA). ELLA+Muon collapses 8.4 points below the leading band, losing task fit rather than retention. The mechanistic core: merged updates under AdamW concentrate in 1.37–1.84 effective singular directions, Muon spreads them over ~7, with no overlap in any tracked run; realized update magnitudes (Muon/AdamW ratios 0.91–2.06) do not explain the differences. Limitations: the IncLoRA baseline sits 12 points above the previously reported value, unexplained; penalty strengths are not re-tuned under Muon; TRACE alone adds step-size matching (with an unverified constant) and early stopping; scale is limited to encoder-decoder T5 models and no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Plain IncLoRA trained with Muon reaches the accuracy band of dedicated CL methods on Standard CL (81.24 vs 81.00/79.64 OA) and beats every AdamW configuration on TRACE (31.36–31.80 vs 24.18–25.92 OA).
- One update-constraining mechanism suffices; stacking a second adds nothing on Standard CL and costs the most restrictive method (ELLA) 8.4 points of accuracy and task fit.
- The optimizer difference is distributional, not magnitudinal: AdamW confines merged updates to 1.37–1.84 effective singular directions while Muon spreads them over ~7, with zero overlap across all 90 tracked runs.
- Realized update magnitudes (Muon/AdamW ratios 0.91–2.06) rule out step size as the explanatory variable for the retention results.
Strengths
Complete, artefact-resistant geometric separation between optimizers
Median concentration σ1/‖ΔW‖F is 0.737–0.854 under AdamW (1.37–1.84 effective directions) vs 0.378–0.379 under Muon (~7 directions) across all six configurations; decile bounds 0.643 vs 0.386 over 90 runs × 83 logged steps (Table 4, Section 4). Appendix C shows the separation survives task boundaries (0.847/0.378 within-task vs 0.813/0.379 in ±10-step windows) and verifies the spectrum is effectively rank-r.
Unusually disciplined claim structure
Section 5's 'What we do not claim' enumerates seven explicit concessions; the authors refuse a ranking at a 0.24-point margin (7/15 paired cells) and use paired per-(seed, order) differences (Table 5) so task-order effects of up to 6.1 points cancel exactly rather than inflating the error term.
Exact failure-mode decomposition
Step-size confound measured, not assumed
Pipeline sanity-checked against prior work
Weaknesses
Unexplained 12-point baseline discrepancy undercuts calibration
TRACE headline wins are confounded by protocol changes applied only there
Penalty hyperparameters never re-tuned under Muon
Mechanistic gap between what is measured and what is claimed
Narrow scale and thin TRACE sampling
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a Workshop-in-Progress paper (CoLLAs 2026) asking a sharply scoped question: is the update restriction that anti-forgetting LoRA methods impose required to be task-aware, or does a generic optimizer-level constraint suffice? The authors train unconstrained incremental LoRA (IncLoRA) with Muon (Jordan et al., 2024), which orthogonalizes each step, and compare six method–optimizer combinations (IncLoRA, O-LoRA, ELLA × AdamW, Muon) on the Standard CL Benchmark (T5-Large, K=4, three ELLA task orders, five seeds; 90 runs) and TRACE (Flan-T5-Large, K=8, Order 7, three seeds). Three claims: (1) IncLoRA+Muon reaches the dedicated methods' band on Standard CL and beats every AdamW configuration on TRACE; (2) one update-constraining mechanism suffices — a second adds nothing and costs ELLA+Muon 8.4 points; (3) the optimizer difference is distributional (1.37–1.84 vs ~7 effective singular directions), not magnitudinal (ratios 0.91–2.06×). The claim structure is unusually disciplined: Section 5's 'What we do not claim' enumerates seven concessions, and the authors explicitly decline a ranking at a 0.24-point margin.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Muon and why would it help continual learning?
Muon is an optimizer that orthogonalizes each weight update via Newton-Schulz iterations, flattening its singular values. This paper argues that acts as a generic update constraint — similar in spirit to the loss penalties O-LoRA and ELLA add — spreading each LoRA update over roughly 7 effective singular directions instead of AdamW's 1–2, which reduces catastrophic forgetting without any task-aware machinery.
Did Muon actually beat O-LoRA and ELLA?
Not exactly — it matched them. On the Standard CL Benchmark, plain LoRA+Muon scored 81.24 overall accuracy versus 81.00 for O-LoRA+AdamW and 79.64 for ELLA+AdamW; the authors say the 0.24-point lead is within seed noise. On TRACE, all three Muon configurations scored above all three AdamW ones (31.36–31.80 vs 24.18–25.92 OA).
Why did ELLA combined with Muon do badly?
Stacking ELLA's loss penalty on top of Muon's orthogonalization applies two constraints at once. ELLA+Muon fell 8.4 points below the leading configurations (71.24 OA) because it stopped fitting each task well (fit 72.7 vs 80.7–83.3 elsewhere) while retention stayed fine. The authors did not re-tune ELLA's penalty strength under Muon, so the combination was evaluated 'as it comes'.
What are the main limitations of this study?
Only T5-Large and Flan-T5-Large encoder-decoder models were tested; TRACE used 3 seeds on a single task order; the plain-LoRA baseline came out 12 points above the previously reported value with no explanation; penalty strengths were never re-tuned under Muon; TRACE comparisons involved a step-size matching rule with an unverified constant plus early stopping; and no code release is mentioned.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 are assessed from captions and in-text descriptions only; the underlying trajectories could not be inspected visually.
- No code or data availability statement appears in the extracted text; codeAvailable=false reflects absence of evidence, not a confirmed negative.
- Reported external baselines (63.6 and 79.9 from Das Biswas et al. 2026) are taken from the paper's own citation and were not independently verified.
- Appendix D's derivation was checked arithmetically from the extracted text; symbol rendering (e.g., square roots in the matching rule) may be imperfect in the source extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.