Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters
Egor Romanyukov, Timofey Novikov, Timur Shokarov, Elizaveta Zorkina, Anastasia Palienko, Stepan Dergachev
cs.LG · cs.CL
Quick verdict
Step Law LR ~4x too high below 3M params; recalibrated power laws fit; promising but single-seed, confounded by protocol changes from the original
Summary
This paper tests whether Step Law (Li et al., 2025), power-law formulas for the optimal peak learning rate and batch size calibrated on 59M–1B parameter models, transfers to small language models. Using a single nanoGPT/TinyStories pipeline with a 2048-token BPE tokenizer, the authors grid-search learning rate and batch size across 29 (N, D) cells (935 runs); the main refit uses 25 cells (815 runs) spanning 0.25–2.03M parameters. Per-cell optima are extracted via a local quadratic fit to the loss surface in log-log coordinates. Verdict: H2 — the power-law form survives but the coefficients differ. Direct transfer overestimates the optimal learning rate by a median factor of about 4 (range 2.4–6.6); the refit gives eta* proportional to N^-0.508 D^0.238 (R^2 = 0.834) and B* proportional to D^0.931 (R^2 = 0.950), with batch size confirmed independent of N (p = 0.87). Main limitations: a single seed (confidence intervals reflect cell-to-cell variation, not seed noise), a narrow tested range, and simultaneous departures from the original setup — tokenizer, corpus, and warmup schedule — so the recalibration coefficients may partly reflect these confounds rather than small scale itself. I also found at least one arithmetically inconsistent row in the main optima table.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Direct transfer of Step Law to sub-59M models systematically overestimates the optimal peak learning rate, by a median factor of about 4.0 (range 2.4-6.6) across the 25 main-refit cells.
- The power-law functional form survives at small scale: eta* is fit by 0.0985 N^-0.508 D^0.238 (R^2 = 0.834) and B* by 3.6e-4 D^0.931 (R^2 = 0.950), so hypothesis H2 is accepted over H1 and H3.
- Step Law's structural claim that the optimal batch size depends only on D and not on N is reproduced in the small-model regime (log-N coefficient p = 0.87; hierarchical F-test p = 0.87).
- The batch-size exponent differs sharply from the original: gamma = 0.931 vs Step Law's 0.571, i.e., B* grows nearly twice as steeply with data budget in this regime.
Strengths
Clear falsifiable hypothesis design
Section 3.3 defines mutually exclusive, jointly exhaustive outcomes (H1 direct transfer / H2 recalibrated power law / H3 no power law) with a pre-specified excess-loss metric; Table 6 maps results to verdicts explicitly.
Genuine coverage of an untested regime
The main grid spans 0.25–2.03M non-embedding parameters, roughly 30–200x below Step Law's 59M lower calibration bound, across 25 (N, D) cells with 4–6 D values per scale (Table 2, Table 8).
Robust optimum extraction instead of raw argmin
Structural claim tested separately from coefficients
Honest, precisely scoped limitations
Weaknesses
Scale attribution confounded by simultaneous protocol departures
Single seed throughout
Arithmetic inconsistencies in Table 8
Underspecified statistical inference
gamma = 0.931 implies optima far outside the searched grid
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed question: does Step Law (Li et al., 2025) — power-law formulas η* = 1.79 N^-0.713 D^0.307 and B* = 0.58 D^0.571 calibrated on 59M–1B parameter models — transfer below that floor? Three pre-registered-style outcomes are defined in Section 3.3: H1 (direct transfer works), H2 (power-law form holds, coefficients differ), H3 (no power law). The tested regime is far outside the calibration range: the main grid covers 0.25–2.03M non-embedding parameters across 25 (N, D) cells (815 of 935 runs), with two N = 5M anchor cells and two D/N > 600 cells held out as diagnostics (Table 2). The verdict, on pooled data, is H2: the functional form survives; the calibration does not.
The headline claims are quantified and mostly proportionate: direct transfer overestimates η* by a median 4.0x (range 2.4–6.6x, Table 3); the refit gives η* ∝ N^-0.508 D^0.238 (R² = 0.834) and B* ∝ D^0.931 (R² = 0.950); B* independence from N reproduces (p = 0.87). The authors scope conclusions to N < 2.03M and flag β as fragile. The weak point is attribution: because tokenizer, corpus, and warmup rule were all changed together with scale (Sections 5.2–5.3), the observed coefficient drift cannot be assigned to scale alone.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper conclude about using Step Law for small language models?
It concludes you should not use Step Law's formulas verbatim below roughly 2M parameters. In its nanoGPT/TinyStories setup, Step Law's prescribed learning rate was about 4x too high at the median (2.4-6.6x range). The power-law shape still holds, but with recalibrated coefficients, and the practical rule of thumb offered is to divide Step Law's learning rate by roughly four.
What is Step Law?
Step Law (Li et al., 2025, arXiv:2503.04715) is an empirical scaling law for training hyperparameters, fitted from about 3,700 runs on 59M-1B parameter models: optimal peak learning rate eta* = 1.79 N^-0.713 D^0.307 and optimal batch size B* = 0.58 D^0.571, with batch size independent of model size.
Does the optimal batch size depend on model size?
No, according to this paper's small-scale experiments. Adding a log-N term to the batch-size regression changed nothing (p = 0.87, hierarchical F-test p = 0.87), reproducing Step Law's structural claim that batch size is set by the data budget. However, the growth with data was steeper than the original: B* proportional to D^0.931 versus the original D^0.571.
What are the main caveats of this study?
Three stand out. (1) Every run used a single random seed, so the reported confidence intervals only capture variation across (N, D) cells, not seed noise. (2) The tokenizer (2048-token BPE vs GPT-2's ~50K), the corpus (TinyStories), and the warmup schedule (5% of budget vs fixed 2000 steps) all differ from the original Step Law setup simultaneously, so the recalibration may partly reflect those changes rather than small scale. (3) The tested range is 0.25-2.03M parameters, far below the 59M boundary, with only two diagnostic 5M anchor cells.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are present only as extracted text fragments; visual details of the loss-surface heatmaps (Fig. 1) and the exponent comparison (Fig. 2) could not be inspected.
- The linked GitHub repository (code, logs, refit scripts) was not audited; reproducibility judgments rely on the paper's description.
- The abstract is truncated mid-sentence in the provided text; conclusions were cross-checked against the body.
- The extraneous GPU-cost section (Table 7, prices in RUB) was retained in the analysis as context for compute feasibility but is peripheral to the scaling-law claims.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.