Subspace Levenberg Marquardt Algorithms in Training Neural Networks
cs.LG · math.OC
Quick verdict
Compares subspace LM (KSLM/HSLM) with LM, SGD, Adam on two toy NN tasks; HSLM best balances speed and accuracy, but the tiny scale limits scalability claims.
Summary
This OPT2026 workshop paper empirically evaluates two subspace variants of the Levenberg–Marquardt (LM) algorithm — Krylov-subspace LM (KSLM) and the author's adaptive hybrid subspace LM (HSLM) — against classical LM, SGD, and Adam on two small neural-network training problems: fitting a damped-oscillation function with a 1-70-40-1 tanh MLP (≈3,000 parameters) on 40,000 noisy samples, and 13-bit parity classification with a 13-25-10-1 MLP over all 8,192 patterns. Each experiment uses 100 trials with a common initialization across optimizers and fixed data/splits. HSLM achieves the best cost–accuracy balance: on regression it converges in 19.3±5.2 iterations and 25.9±7.4 s with 100% of trials reaching the noise floor, versus 83.0±26.2 s for full LM and 132.8±40.5 s for KSLM — slower than full LM due to Krylov construction overhead; on parity it needs 37.0±27.2 iterations for 100% convergence and 99.6±0.3% validation accuracy, comparable to Adam's 99.8±0.1% at far fewer epochs. Main limitations: the scalability motivation is never tested at a scale where full LM is impractical; SGD/Adam use single untuned learning rates; no L-BFGS baseline; no significance testing; no code release; and the limited-data experiment (Appendix C) is reported only qualitatively.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- HSLM achieves the best balance of convergence speed, computational cost, and accuracy among the five optimizers on both tasks (Tables 1–2).
- LM-based methods reach the noise floor in 100% of regression trials, versus 41% for SGD and 0% for Adam.
- On 13-bit parity, HSLM converges in far fewer iterations (37.0±27.2) than SGD (1061.7±530.1) and Adam (333.3±458.5) with a 100% convergence rate.
- Subspace formulations can preserve LM-like fast convergence while substantially reducing cost (HSLM 25.9±7.4 s vs LM 83.0±26.2 s on regression).
- HSLM remains effective under limited training data (Appendix C, reported qualitatively).
Strengths
Controlled trial protocol
100 trials per experiment with common initialization across all five optimizers, a single fixed data realization (regression) and fixed stratified 90/10 split (parity), isolating initialization sensitivity (Appendix A).
Transparent negative result for KSLM
Table 1 shows KSLM at 132.8±40.5 s is slower than full LM at 83.0±26.2 s on regression; §3.1 openly attributes this to Krylov construction/expansion overhead rather than hiding it.
Both wall-clock and iteration counts reported
Full implementation disclosure
Weaknesses
Scale–claim mismatch
Untuned first-order baselines
Missing L-BFGS baseline
No inferential statistics
Appendix C is qualitative only
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This OPT2026 workshop paper (arXiv:2609.00789) is an empirical evaluation, not a new algorithm. The sole author — who introduced HSLM in a companion preprint (ref. [5], arXiv:2608.25524) — compares five MLP-training optimizers: classical Levenberg–Marquardt (LM), Krylov-subspace LM (KSLM, after Mizutani & Demmel 2001), hybrid subspace LM (HSLM), SGD, and Adam. Two tasks: (i) regression of f(x)=2e^(−x²)cos(2πx) on [−2,2] with 40,000 noisy samples (σ_noise=0.05σ_y) using a 1-70-40-1 tanh MLP (≈3,000 parameters); (ii) 13-bit parity over all 8,192 patterns with a 13-25-10-1 tanh MLP (≈620 parameters). Central claims: LM-family methods converge in far fewer iterations than SGD/Adam; HSLM achieves the best time–accuracy balance; subspace formulations can extend LM's practical advantages to larger problems (§4). The framing is a scalability argument, and that is where claims and evidence diverge most.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the HSLM optimizer?
Hybrid subspace Levenberg–Marquardt: at each iteration it builds a low-dimensional subspace from the normalized gradient, recent optimization steps, Lanczos/Krylov vectors, and randomized Gauss–Newton probes; it checks that the subspace captures at least 99% of the gradient energy, then solves a small damped LM system inside that subspace using curvature-dependent damping, with Armijo backtracking for step acceptance.
Is subspace LM faster than regular Levenberg–Marquardt for training neural networks?
In this paper's two small benchmarks, the hybrid variant (HSLM) was faster: 25.9 s vs 83.0 s on a regression task and 3.7 s vs 5.8 s on 13-bit parity. However, the Krylov variant (KSLM) was slower than full LM on regression (132.8 s vs 83.0 s) because constructing the Krylov basis adds overhead, so 'subspace' does not automatically mean cheaper.
Did subspace LM beat Adam?
Not on final accuracy: on 13-bit parity Adam reached the best validation accuracy (99.8%) versus HSLM's 99.6%. HSLM's advantage was speed — about 37 iterations versus Adam's 333 epochs on average, with a 100% convergence rate versus 92% for SGD and LM. On the noisy regression task, Adam never reached the noise-floor error target within its 1500-epoch budget in this particular setup.
Is code or data available for this paper?
The paper states no code or data release. However, the appendices fully specify network architectures, all hyperparameters, stopping criteria, and the laptop hardware used, so the experiments are CPU-scale and straightforward to reimplement in NumPy.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels (Figs 1–5) are embedded images; only captions and in-text summaries were extractable, so curve shapes could not be inspected — all quantitative claims here come from the in-text Tables 1–2 and Appendix A.
- Equation typesetting is garbled in places by PDF extraction (e.g., the min-operator line in §2.1), but all formulas were reconstructable from context and no content appears missing; the text ends naturally at Appendix C.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.