HyperMC: Multi-Fidelity Hyperparameter Tuning for Stochastic Gradient MCMC
stat.ML · cs.LG · stat.CO
Quick verdict
Hyperband+KSD multi-fidelity tuning for SGMCMC, with robust variant and budget theory; a solid MAMBA extension, but KSD is both tuning target and main metric.
Summary
This paper proposes HyperMC, a multi-fidelity framework for tuning stochastic gradient MCMC (SGMCMC) hyperparameters — step size, mini-batch size, and leapfrog steps — in the absence of Metropolis–Hastings acceptance rates. It wraps Hyperband's multiple successive-halving brackets (η=3, wall-clock time as resource) around kernel Stein discrepancy (KSD) evaluation, sampling step sizes continuously instead of from MAMBA's fixed discrete grid. Robust HyperMC replaces independent bracket initialization with a deterministic global grid followed by elite-guided local refinement whose radius halves each bracket, at no extra sampler evaluations. Under a deterministic finite-resource envelope and sub-Gaussian concentration of KSD scores, the authors prove the successive-halving component returns a near-optimal configuration among sampled candidates with probability 1−δ, with an explicit sufficient budget (Theorem 4, proofs in Appendix B). Experiments on synthetic logistic regression (N=10^6), probabilistic matrix factorization (MovieLens 100K), and a Bayesian neural network (MNIST), each across six SGMCMC samplers, show selected configurations achieve the lowest or near-lowest KSD (Tables 3, 6, 9) and better ECE/MCE calibration, while Robust HyperMC narrows run-to-run variability. Key limitations: KSD is both the tuning objective and the headline metric, with mixed independent posterior-SD results (Table 3 SGHMC: 91.8 vs grid search 38.9); main tables report no uncertainty; no code release is mentioned; wall-clock budgets impede exact replication.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Hyperband-style multi-bracket allocation with continuous step-size sampling and KSD ranking (HyperMC) selects SGMCMC configurations with the lowest or near-lowest KSD across logistic regression, PMF and a BNN, beating MAMBA, predictive grid search and the h=1/N heuristic (Tables 3, 6, 9).
- Robust HyperMC's global-grid initialization plus elite-guided refinement reduces run-to-run variability of tuning outcomes without any additional sampler evaluations (Figures 2 and 4).
- Under a deterministic finite-resource envelope and sub-Gaussian concentration of KSD scores, the successive-halving component returns a configuration within epsilon/2 of the best sampled candidate with probability at least 1-delta, given the explicit budget condition of Theorem 4.
- KSD-based tuning translates into better predictive calibration (smallest or tied-smallest ECE and favorable MCE) for the MNIST Bayesian neural network (Table 9), supporting posterior-discrepancy criteria over predictive accuracy alone.
Strengths
Well-scoped extension of MAMBA with continuous search
HyperMC keeps MAMBA's thinned full-gradient KSD estimator (Sections 3.5.2-3.5.3) so comparisons isolate the allocation strategy; continuous log-step sampling (log10 h in [-8,-1] or [-8,-2], Tables 1/4/7) removes MAMBA's grid-resolution dependency, and multiple brackets trade exploration against evaluation fidelity.
Theory with complete proofs and explicit assumptions
Lemmas 1-2, Definition 3 and Theorem 4 are proved in full in Appendix B, including the union-bound uniform event and the geometric-schedule budget argument; assumptions (deterministic fidelity envelope, sub-Gaussian KSD scores, visible via the mgf bound in the B.1 proof) are stated rather than hidden.
Validation beyond the tuning metric
Candor about mixed and negative results
Practical guidance for practitioners
Weaknesses
Evaluative circularity
Resource abstraction mismatch
No uncertainty quantification
Selection-rule inconsistency
Reproducibility gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
HyperMC: Multi-Fidelity Hyperparameter Tuning for Stochastic Gradient MCMC — Structured Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is HyperMC and what problem does it address?
HyperMC is a multi-fidelity hyperparameter tuning framework for stochastic gradient MCMC (SGMCMC) samplers such as SGLD, SGHMC and SGNHT. These samplers lack a Metropolis-Hastings acceptance rate, so classic acceptance-based tuning does not apply; HyperMC instead runs Hyperband-style successive-halving brackets that allocate wall-clock-time budgets across candidate configurations (step size, mini-batch size, leapfrog steps) and ranks them by kernel Stein discrepancy (KSD), a sample-quality measure requiring no normalizing constant.
How does HyperMC differ from MAMBA?
MAMBA (Coullon et al., 2023) also ranks SGMCMC configurations by KSD under successive halving, but picks step sizes from a fixed discrete grid under a single resource-allocation schedule. HyperMC samples step sizes from a continuous interval and runs multiple Hyperband brackets with different exploration-versus-evaluation trade-offs; a Robust variant additionally initializes the first bracket from a global grid and refines around elite configurations. The paper reports lowest or near-lowest KSD for HyperMC-selected configurations on logistic regression, MovieLens 100K matrix factorization, and an MNIST Bayesian neural network.
Does the theoretical guarantee apply to real SGMCMC runs?
Only conditionally. Theorem 4 assumes a deterministic bound on how much KSD improves with more compute (a fidelity envelope) and sub-Gaussian concentration of i.i.d. KSD score contributions — conditions that correlated MCMC output does not literally satisfy, and whose constants (true performance gaps, sigma squared) are unknown in practice. It is best read as a qualitative justification of successive halving under noisy finite-budget KSD estimates, not as a practical stopping or budgeting rule.
What data and code are used in the paper?
Experiments use a synthetic logistic-regression dataset with one million observations, the public MovieLens 100K ratings dataset, and MNIST — all standard — plus six SGMCMC samplers and a NumPyro NUTS reference posterior. The provided text mentions no code or data release, and budgets are expressed in wall-clock seconds on unspecified hardware, so exact replication of the reported numbers would require reimplementation and re-tuning of hardware-dependent settings.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extraction skipped three middle sections ([...sampling gap...]); statements that run counts, hardware, thinning ratios and code availability are 'unstated' are based on the visible text and could in principle be addressed in unseen sections.
- The formal statements of Assumptions 1-2 fall inside a sampling gap; this review characterizes them from their use in Lemma 2, Theorem 4 and the visible Appendix B proofs (sub-Gaussian mgf bound; deterministic envelope |l_r(x) - nu(x)| <= gamma(r)).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.