Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasets
cs.LG
Quick verdict
发现 L1 求解器相关文献仅在极端惩罚下进行测试；旧的 Liblinear 和简单的 LBFGS* 基线在实际 C 值下胜出；但计时混合了停止容差的影响。
Summary
论文认为，关于 L1 惩罚逻辑回归求解器的研究在评估进展时，所处于的机制与实践脱节：基于 λmax 的惩罚在数百万特征中仅能选出约 100 个非零特征，而 Cmin（零解惩罚）比常见的默认值 C=1 低约 5 个数量级。作者调研了 skglm、Celer、Blitz、fireworks 和 benchopt 的评估实践，建议报告 C ∈ [10^-2, 10^2]、非零计数和准确率。在七个公开数据集（最大达 1.496 亿行、5470 万特征、11 GB）以及 gisette 数据集上，他们对比了 skglm、Celer、Celer-PN、SAGA、Liblinear、OWL-QN 以及他们提出的 LBFGS* ——即在最小范数次梯度上应用 L-BFGS，并结合基于 Brent 的初始化、EMA 停止规则、线搜索失败后清除历史记录以及定期剪枝活动集。在 C=0.01 和 C=1 时，Celer 和 SAGA 常常超过 24 小时；Liblinear 最快且综合表现最好；LBFGS* 在 14 种设置中的 8 种达到了最佳测试准确率，且通常比 OWL-QN 更快（11/14）。Liblinear 的多核模式在 12 核时慢了 2 倍以上，而简单的 LBFGS* 并行化则有所助益。生产环境部分描述了 30 TB 以上的分布式恶意软件训练。局限性：不等的停止容差混淆了计时结果，LBFGS* 的目标函数值通常较差，单次运行缺乏误差条，且未说明代码发布情况。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The L1-solver literature anchors evaluation at λmax-derived penalties that select roughly 100 or fewer features out of millions, a regime practitioners do not use.
- At realistic penalties (C=0.01, C=1), the 2012-era Liblinear newGLMNET coordinate-descent solver is the fastest and best all-around method, while recent solvers (skglm, Celer, Celer-PN, SAGA) often fail to converge within 24 hours.
- LBFGS applied to the minimum-norm subgradient with simple engineering heuristics (LBFGS*) is competitive — best test accuracy in 8 of 14 dataset/penalty pairs and faster than OWL-QN in 11 of 14 — despite lacking any convergence guarantee.
- Liblinear's multicore mode is at least 2× slower than its single-core mode at p=12 on every test, whereas naive data-parallel LBFGS* reliably speeds up.
- Solver papers should report absolute C values, solution non-zero counts, and test accuracy instead of λ/λmax ratios alone.
Strengths
Well-evidenced diagnosis of a broken evaluation regime
Table I shows the zero-solution penalty Cmin for all seven datasets ranges from 3.52e-8 (kdd12) to 4.55e-5 (webspam), i.e., ~5 orders below the common default C=1; combined with Sec. II-B's survey (skglm on kdda at λ/λmax=10^-3 ⇒ C=0.0001; Celer at λmax/20; benchopt at λmax/100), the claim that prior work tests ≤100-nonzero solutions out of millions is concretely documented.
Representative baseline coverage with a built-in ablation
Sec. II-A spans coordinate descent (Liblinear/newGLMNET, skglm, Celer ± ProxyNewton), variance-reduced gradients (SAGA), and quasi-Newton (OWL-QN); the Celer vs Celer-PN pair isolates one design component, and exclusions (feature-doubling methods, ReHLine lacking logistic loss) are stated with reasons.
LBFGS* is fully specified, not hand-waved
Practitioner-relevant parallelism finding
Honest scoping of the method's weakness
Weaknesses
Timing comparisons are confounded by unequal stopping criteria
Solutions at the same C are not equivalent, and the paper only partially confronts this
Recommendation–practice mismatch
No statistical rigor
No code release stated; production claims unverifiable
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an empirical benchmark and position paper on solvers for L1-penalized logistic regression (Eq. 1, elastic-net form with logistic loss), motivated by malware classification on byte n-grams where d exceeds one million. Two central claims: (1) the solver literature evaluates regularization anchored at λmax — the weakest penalty yielding w=0 — and therefore tests solutions with ≲100 non-zeros out of millions of features, a regime practitioners never use; (2) at realistic penalties (C=0.01 and C=1), a 2012-era coordinate descent method (Liblinear/newGLMNET) remains the best all-around solver, and a deliberately 'unprincipled' LBFGS* baseline — L-BFGS on the minimum-norm subgradient with four engineering heuristics — is competitive despite lacking a convergence guarantee. Table I supports the regime claim concretely: Cmin spans 3.52·10^-8 (kdd12) to 4.55·10^-5 (webspam), roughly five orders of magnitude below C=1, while surveyed prior work tops out near C=Cmin·1000 (skglm on kdda at λ/λmax=10^-3 ⇒ C=0.0001; Celer at λmax/20; benchopt at λmax/100).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LBFGS* in this paper?
A deliberately simple baseline introduced by the authors: run standard L-BFGS on the minimum-norm subgradient of the L1-penalized logistic loss, plus four engineering tweaks — a one-dimensional Brent initialization, an exponential-moving-average stopping rule, clearing the quasi-Newton history when a line search fails, and periodically pruning features whose gradient falls below the penalty threshold. It has no convergence proof, but in the paper's benchmarks it matched purpose-built solvers on test accuracy at C=0.01 and C=1.
Why do the authors say L1 solver papers evaluate the wrong regularization range?
Most prior work expresses the penalty as a ratio of λmax — the weakest penalty that drives all weights to zero — and tests ratios like λmax/20 to λmax/1000. The paper's Table I shows the corresponding C values are roughly five orders of magnitude below the common default C=1, selecting about 100 features out of millions — far sparser than what practitioners use or what maximizes accuracy.
Which solver was fastest for L1 logistic regression in this study?
LIBLINEAR's newGLMNET coordinate descent, a method from around 2012, was fastest and most consistent across seven large datasets at C=0.01 and C=1. Recent solvers (Celer, Celer-PN, SAGA) often failed to converge within 24 hours at C=1, and LIBLINEAR's own multicore mode was at least 2× slower than its single-core mode when using 12 cores.
Does L-BFGS converge on L1-penalized problems?
Not provably — the L1 term is non-differentiable at zero, which is why the OWL-QN algorithm was developed. The authors argue empirically that this theoretical obstacle matters little at industrial scale: their heuristic LBFGS* variant was competitive in accuracy and runtime across all tested public datasets.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table II/IV extraction misaligns or drops cells in several C=1 rows (webspam, avazu-app/site, kdda, kddb, kdd12); per-solver attributions in those rows are uncertain, and the 'red = failure' color coding is unrecoverable from text.
- Figures 1–2 (active-set sparsity over iterations; cumulative time) are available only as captions and could not be verified.
- The appendix code listing is truncated/garbled in places (e.g., broken modulo expressions at history-buffer updates), so some implementation defaults could not be confirmed.
- Core directional results (timeout patterns, Liblinear dominance, 8/14 accuracy, 11/14 vs OWL-QN, ≥2× multicore slowdown) are stated consistently in prose and timed rows, but many exact table cells could not be cross-checked — hence medium confidence.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.