Model-Agnostic Feature Selection via LOCO-Guided Adaptive Minipatch Sampling
stat.ML · cs.LG · stat.ME
Quick verdict
LAMPS: adaptive minipatch LOCO feature selection with exact-recovery theory; sound and novel, but assumption-heavy, no FDR control, modest-scale evidence.
Summary
The paper introduces LAMPS (LOCO-guided Adaptive Minipatch Sampling), a model-agnostic feature selection framework for regression. LAMPS repeatedly trains a black-box base learner on minipatches (subsampled observations and features), computes leave-one-covariate-out (LOCO) importance scores without refitting via model averaging, and updates feature sampling probabilities with a capped-reweighting rule that exploits high-importance features while capping probabilities at delta to preserve exploration. The final feature set is obtained by thresholding sampling probabilities at delta/2, following the stability-selection majority principle. Theoretically, the authors prove exact support recovery after a logarithmic number of iterations in high-dimensional sparse additive models, provided base predictors are well trained on average (Theorems 1-2), extend the guarantee to models with one two-feature interaction (Theorem 3), and give an illustrative analysis of correlated features (Proposition 1). Experiments on synthetic linear, nonlinear additive, and interaction models (e.g., N=200, M=500) show higher F1 scores than Lasso, Elastic Net, stability selection, CPSS, knockoffs, MARS, and SpAM, especially under correlated features; case studies on riboflavin (N=71, M=4088) and ROSMAP data support the method. Limitations include assumption-heavy theory (e.g., Assumption 8 on data-independent sampling distributions), absence of FDR control, unreported computational costs (K scales with M-squared), and moderate simulation scale with 10 replicates.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LAMPS performs feature selection with any black-box regression base learner that need only produce predictions, via LOCO-guided adaptive minipatch sampling with capped reweighting (Alg. 1-2).
- Under sparse additive models with independent features and base predictors that are well trained on average, LAMPS achieves exact support recovery within Theta(log(M/m)) iterations with probability at least 1-(M/N)^-c, including regimes where M exceeds N (Theorem 2).
- The exact-recovery guarantee extends to features that contribute only through one pairwise interaction, at the cost of a stronger signal condition scaled by 1/(q_u ∧ q_v) (Theorem 3).
- The capped sampling probability (max delta=0.8) prevents a correlated proxy from fully replacing a signal feature, which explains empirically superior behavior under feature correlation (Section 3.4, Proposition 1, Figures 3-6, Table 1).
- LAMPS outperforms state-of-the-art feature selection baselines on synthetic and real data, with the largest gains in correlated settings.
Strengths
Genuinely novel adaptive update for continuous importance scores
Alg. 2's capped-reweighting (shift scores by -min+c0/M, truncate weights at t* so max_j q_j <= delta) is designed for unbounded, possibly negative LOCO scores, unlike prior adaptive schemes (Thompson VS, RaSE, AdaSTAMPS/AdaSub) that consume binary selection events; Section 2 and Appendix A make this contrast explicit.
Substantive theoretical contribution with new proof machinery
Theorem 2 requires uniform-in-j control of relative LOCO error eps_j/q_j over all M features and all B iterations — beyond Gan et al. (2026) who bound one feature under uniform sampling. New tools: fine-grained error decomposition (Lemma S6), smoothness of importance in q via an explicit coupling (Lemmas S9/S12, Alg. S4), and a read-k Chernoff bound for weakly dependent averages (Lemma S14). The geometric growth q_j^(b) >= 1.5 q_j^(b-1) (additive case) yields the log-iteration result; interactions are covered with the expected 1/(q_u ∧ q_v) co-occurrence penalty (Theorem 3, Assumption 11).
Computational elegance
Convincing correlated-feature behavior on the riboflavin benchmark
Complete, reproducible experimental specification
Weaknesses
The exact-recovery theorem leans on a stack of unverifiable assumptions
No error control on the selected set
Computational cost unreported despite M-squared scaling in K
Closest adaptive competitors absent from the empirical comparison
Modest empirical scale and illustrative-only correlation theory
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a methodology paper in high-dimensional statistics and machine learning. The authors propose LAMPS (LOCO-guided Adaptive Minipatch Sampling), a model-agnostic feature selection framework for regression that wraps any black-box base learner (demonstrated with OLS, Lasso, MARS, SpAM). Three central claims: (i) LAMPS converts the essentially free LOCO importance available from minipatch ensembles (building on Gan et al. 2026, LOCO-MP) into a direct selection procedure via adaptive feature subsampling; (ii) under sparse additive models with independent features and base predictors that are well trained on average, LAMPS achieves exact support recovery within Theta(log(M/m)) iterations, including when M exceeds N (Theorems 1-2, probability at least 1-(M/N)^-c), with an extension to features entering only through one pairwise interaction (Theorem 3); (iii) empirically it outperforms Lasso, Elastic Net, stability selection, CPSS, knockoffs, MARS, SpAM, and HSIC-Lasso, particularly under correlated features (Figures 4-6; Table 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LAMPS and how does it select features?
LAMPS is a feature selection method for regression that repeatedly trains any black-box base learner on small random subsets of both samples and features (minipatches). After each round it computes leave-one-covariate-out (LOCO) importance cheaply by averaging the already-trained models, raises the sampling probability of high-importance features (capped at delta, typically 0.8, to keep exploring), and finally selects features whose final sampling probability exceeds delta/2 (about 0.4).
Does LAMPS provide false discovery rate (FDR) control like knockoffs?
No. LAMPS offers exact-recovery guarantees under its theoretical assumptions, but the selected set carries no finite-sample FDR or Type-I error control; the delta/2 threshold follows the stability-selection majority principle. If calibrated FDR control is required, knockoffs remain the standard option, at the cost of needing access to the joint feature distribution.
Which base models can be used with LAMPS?
Any regression algorithm that produces predictions — the paper demonstrates ordinary least squares, Lasso (on the riboflavin gene data), MARS and SpAM (on nonlinear simulations and ROSMAP Alzheimer's data). The base learner does not need to perform feature selection itself, which is the main difference from wrapper-style adaptive methods.
Is code available for LAMPS?
Yes. The authors state that software implementing LAMPS and scripts reproducing all empirical results are available on GitHub at github.com/liu-xuhui/lamps.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction omits the statement of Theorem 1, Assumptions 4-7, parts of Sections 3.1-3.2, the end of Section 5.2 (ROSMAP metric), and later appendix sections; this review reflects only the visible text.
- Figures 1-6 and Table 1 are image-based; conclusions about them rest on captions and in-text descriptions, not the graphics.
- The abstract is truncated mid-sentence in the supplied text.
- Some theoretical conclusions rely on lemma statements (e.g., Lemma S1's 1.5 growth constant) that appear only in partially omitted sections.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.