Scaling of Capability and Efficiency at Inference Time in Large Reasoning Models
Moritz Laber, Zohair Shafi, Germans Savcisens, Brennan Klein, Matteo Chinazzi, Samuel V. Scarpino, Albert-László Barabási, Alessandro Vespignani, et al.
cs.LG · physics.soc-ph
Quick verdict
Claims LRM capability scales sublinearly but efficiency does not; rigorous Bayesian evidence on synthetic tasks, limited to one distilled model family.
Summary
The paper asks whether scaling up large reasoning models improves both capability and token efficiency at inference time. The authors evaluate five DeepSeek-R1-Distill models (1.5B to 70B parameters; only the 70B is Llama-based, the rest Qwen-based) on four synthetic, contamination-free reasoning tasks — addition, bracket nesting, parity, and indexing — with instance sizes up to 500, 100 instances per size, and three sampling temperatures. Hierarchical Bayesian models relate correctness and output length to instance size and parameter count, with model comparison via leave-one-out expected log predictive density. Three findings emerge: the probability of solving an instance decays approximately exponentially with size; the decay scale grows sublinearly with parameters (posterior-mean scaling exponents 0.07–0.75 across all 24 task-by-temperature conditions; e.g., 0.69 with 95% credible interval 0.53–0.85 for addition at τ=0.6, the scale ν rising from ≈26 at 1.5B to ≈364 at 70B); and output length follows a power law in instance size whose prefactor shows no systematic dependence on model size. Main limitations, acknowledged by the authors: a single distilled model family, only five size points, and toy algorithmic tasks.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The probability that a DeepSeek-R1-Distill model solves a task instance decays approximately exponentially with instance size across four arithmetic/algorithmic tasks.
- The characteristic decay scale of this decline grows sublinearly with parameter count (posterior-mean exponents 0.07–0.75 in all 24 task-by-temperature conditions).
- Output length of correct responses follows a power law in instance size with exponents 0.41–0.89, i.e., sublinear in n.
- The prefactor of the output-length power law does not vary systematically with model size, providing no evidence that larger models are more token-efficient.
Strengths
Rigorous hierarchical-Bayesian pipeline
Explicit likelihoods (binomial for correctness; log-normal with Taylor-derived variance in Appendix A), fully listed priors (Eqs. 16–24), NUTS with 6 chains, R-hat < 1.01, no divergences, and LOO-elpd comparison with full refits (Sec. IV E, Table A5) — well above typical LLM evaluation practice.
Contamination-free, verifiable tasks
Four synthetic tasks with programmatic ground truth and tunable size n (Sec. IV B, Table I); the brackets generator (p_open = 0.6, p_false = 0.5) is given in pseudocode, and Appendix D checks the plausible length-shortcut via the appearance of the term stack in answers.
Capability result is robust across conditions
Honest scoping and hedging
Weaknesses
Scaling exponent estimated from five points
Base-architecture confound on the size axis
Efficiency null has limited statistical power
Token count as the only efficiency metric
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper studies two latent properties — capability (probability of solving an instance) and efficiency (tokens per correct solution) — across five DeepSeek-R1-Distill models (1.5B/7B/14B/32B Qwen-based; 70B Llama-based) and four synthetic tasks (addition, Dyck-3 brackets, parity, index), with instance sizes n ∈ {2, 10, 50, 100, ..., 500}, R = 100 instances per size, at temperatures τ ∈ {0.4, 0.6, 0.8}. Four claims are made: (1) solve probability decays approximately exponentially in n; (2) the decay scale ν grows sublinearly with parameter count; (3) output length among correct responses follows a power law in n; (4) the prefactor of that power law shows no systematic dependence on model size. The claims are appropriately scoped and hedged; the abstract's framing of 'potential limitations of naive scaling' stays within the evidence.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do larger reasoning models use fewer tokens to solve problems?
In this study of five DeepSeek-R1-Distill models (1.5B–70B) on four synthetic tasks, no: the token cost of correct answers followed a power law of problem size whose prefactor did not systematically shrink with model size. Larger models solved much larger instances, but were not measurably more token-efficient on instances they both could solve.
What is the scaling exponent between DeepSeek-R1-Distill model size and reasoning capability?
Capability was measured as the characteristic decay scale of the solve probability with problem size. Its power-law exponent on parameter count had posterior means of 0.07–0.75 depending on task, temperature, and model variant (e.g., 0.69 for addition at the recommended temperature) — below 1, meaning diminishing returns: doubling parameters raises the solvable-size scale by only about 1.6×.
Is the paper's code and data available?
Yes. The tasks are synthetically generated with fully specified generators, model checkpoints are public, and the authors state that generation and analysis code is at github.com/zohairshafi/capability-efficiency-scaling. Re-running inference on the largest models requires datacenter GPUs (H100/H200 were used); the statistical refits are CPU-scale.
Do these findings apply to GPT, Claude, or Gemini?
Not directly. The study covers only one distilled open-weight model family on toy arithmetic and algorithmic tasks. The authors list generalization to other model families, other training pipelines, and harder naturalistic benchmarks as open questions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (Fig. 1–5, A1) could not be inspected from the extracted text; fit-quality judgments rest on captions, tables A1–A5, and reported elpd values.
- The extraction contains layout artifacts (subscripts, equation fragments), though all appendix tables were recoverable; some per-instance details (raw traces, per-temperature decay plots) are not verifiable from text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.