Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search
Zhiliang Chen, Sebastian Ament, David Eriksson, Maximilian Balandat, Bryan Kian Hsiang Low, Eytan Bakshy, Jihao Andreas Lin
cs.LG · cs.AI
Quick verdict
PLES cuts LLM scaling-law estimation cost via entropy-search BO with strong empirical gains, but evaluation is narrow and the closest baseline is untested.
Summary
The paper introduces Power-Law Entropy Search (PLES), a cost-aware acquisition function for multi-fidelity Bayesian optimization that estimates optimal hyperparameter scaling laws for LLM training. Instead of optimizing a single objective, PLES selects at each iteration the experiment (model size N, data size D, hyperparameter θ) that maximally reduces the differential entropy of the coefficients of an assumed power-law θ*=cN^α D^β, per unit computational cost (~N·D). Coefficients are estimated by Thompson sampling from a GP surrogate of the loss to obtain per-scale optima, followed by Bayesian linear regression in log-space, with a natural stopping criterion based on posterior uncertainty at the held-out scale. On a synthetic benchmark, a GP surrogate fitted to published LLM grid-search data (Li et al., 2025), and real Llama-3-style pre-training runs (N ∈ [9.4M, 600M], 20 tokens-per-parameter, learning-rate tuning only), PLES reaches accurate held-out-scale optima in under one-tenth of the compute of grid search, Ladder BO, and Sobol baselines (10 seeds each). The paper also reports that tuning only the learning-rate law recovers the batch-size law as a byproduct. Limitations: the power-law functional form is assumed; the real-data experiment covers a single hyperparameter and one scaling dimension; the closest related method (Li et al., 2026) is not compared empirically; no code release is mentioned; and key method hyperparameters (cost-cooling factor d, number of Thompson samples) are unspecified.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PLES selects experiments maximizing the reduction in differential entropy of power-law scaling-law coefficients per unit compute cost (Eq. 9).
- Acquisition optimization in PLES can be solved by drawing Thompson samples, which converge to the true solution with sufficient samples (asserted, not proven).
- Across synthetic, surrogate, and real LLM pre-training settings, PLES converges to accurate held-out-scale optima using less than one-tenth of the computational budget of grid search, Ladder BO, and Sobol (Figure 4, 10 seeds each).
- Acquiring only to reduce learning-rate scaling-law uncertainty recovers an accurate batch-size scaling law as a byproduct (Figure 6, Section 5.5).
- Posterior uncertainty of the coefficients at the held-out scale provides a practical stopping criterion (Section 4.4, Figure 5).
Strengths
Novel problem reframing
The acquisition function (Eq. 9, Section 4.2) targets reduction of differential entropy of the scaling-law coefficient posterior per unit compute (ND)^d, rather than optimizing a single objective — a genuinely different objective from standard BO, well motivated in Sections 1-2.
Practical, principled stopping criterion
Section 4.4 uses posterior variance x_max^T Σ_w x_max at the held-out scale; Figure 5 shows this quantity becomes small by iteration 6 of the surrogate experiment and further iterations add nothing, demonstrating the criterion works as claimed.
Validation on real training runs
Strengthened grid-search baseline
Candid limitations section
Weaknesses
GP-as-oracle circularity in surrogate experiments
Narrow real-data scope
Missing empirical comparison to closest prior work
Under-specified method and no code release
Thin statistical reporting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets optimal hyperparameter scaling laws — how the loss-minimizing hyperparameter θ* varies with model size N and data size D, assumed to follow the power law θ* = cN^α D^β (Eq. 3), following Li et al. (2025). The claimed contributions are: (1) PLES, a cost-aware acquisition function that selects experiments maximizing reduction of the differential entropy of the power-law coefficient posterior per unit compute (Eq. 9); (2) a Thompson-sampling solution to the acquisition optimization with an asserted asymptotic convergence property; (3) empirical superiority over grid search, per-scale 'Ladder BO', and Sobol sampling across synthetic, surrogate, and real LLM pre-training settings, with the headline that PLES reaches accurate held-out-scale optima in less than one-tenth of the compute budget (Section 5.3, Figure 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Power-Law Entropy Search (PLES)?
PLES is a Bayesian-optimization acquisition function that decides which LLM training experiment to run next so as to maximally reduce the uncertainty (differential entropy) of the coefficients of an assumed power-law θ*=cN^α D^β describing how optimal hyperparameters scale with model and data size, per unit of compute cost. It naturally favors cheap small-scale experiments that are informative about the full scaling law.
How much compute does PLES save compared to grid search?
The authors report that across a synthetic benchmark, a surrogate fitted to published LLM grid-search data, and real Llama-3-style pre-training runs, PLES converged to accurate held-out-scale optima in less than one-tenth of the computational budget required by grid search, per-scale Bayesian optimization (Ladder BO), and Sobol baselines, with 10 random seeds per method.
Does PLES require assuming the scaling law is a power law?
Yes. PLES assumes optimal hyperparameters follow θ*=cN^α D^β. The authors acknowledge in the limitations section that if the true trend deviates from a power law, the Bayesian linear regression step would produce biased coefficients and degrade extrapolation accuracy at the target scale.
Is code available for PLES?
The paper text does not mention any code or data release. The method is built on BoTorch (batched pathwise sampling), and pseudocode is provided in Appendix A, but implementation details such as the cost-cooling factor d and the number of Thompson samples are not specified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Only figure captions are extracted (no figure pixels); the quantitative convergence results live in Figures 4-7, so specific error percentages beyond the qualitative 'less than one-tenth of the budget' claim cannot be verified from the text.
- Mathematical notation contains PDF-extraction artifacts (e.g., stacked fractions in Appendix C render as '(0.1896, , −0.734, , 0.342)'); the sentences themselves are complete — this is a formatting artifact, not truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.