Ladders of Thought: A Self-Evolving Curriculum of Progressively Simplified Reasoning Traces
Minghui Liu, Thomas Magelinski, Dehao Yuan, Qi Yu, Furong Huang
cs.AI · cs.LG
Quick verdict
LoT: rewritten easier question variants + bandit curriculum beat KD for 1-8B LLMs; OOD gains look real but stem mostly from rewrites, not the scheduler.
Summary
The paper introduces Ladders-of-Thought (LoT), a training framework for small and mid-scale LLMs (1–8B) combining three components: progressive question rewrites, where GPT-5-mini replaces premises with their entailed intermediate conclusions to yield semantically faithful, monotonically easier variants; difficulty labeling by minimal reasoning-step count; and a non-stationary multi-armed-bandit scheduler allocating training across step-count buckets via validation-gain rewards. Models are CoT-distilled from GPT-5-mini on GSM8K and EntailmentBank and evaluated on four out-of-distribution benchmarks per domain. Reported pass@5 gains are largest on OOD arithmetic (OPT-2.7B: AddSub 8.26→40.37, +32.11pp; SVAMP 19.06→44.15, +25.09pp) while in-domain gains are small (GSM8K +2–3pp; Pythia-2.8B −0.45); multi-hop results are mixed, with regressions on OpenBookQA and MuSiQue (OPT-2.7B −19.6pp). Ablations favor moderate rewrite depth (≤3 best at 46.09 average vs 42.55 for full ladders) and show step-based difficulty outperforms success-rate bucketing (74.60 vs 63.68 GSM8K on Qwen2.5-7B). Main concerns: Table 12 shows flat sampling over rewrites already averages 37.60, so most of the gain over KD (≈28.2) comes from the rewrites rather than the scheduler (~5pp delta, single configuration); in-domain deltas fall within reported standard errors and no significance tests are given; the lenient four-stage answer verification is unvalidated; and rewriter quality is a cliff-edge dependency (Qwen-72B rewriter: 11.75 GSM8K). Code is promised, not yet released.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reasoning problems can be automatically rewritten into semantically faithful, monotonically easier variants by replacing premise sets with their entailed intermediate conclusions (§3.1).
- Minimal reasoning-step count is a better difficulty signal for reasoning curricula than empirical success-rate bucketing (Table 15: 74.60 vs 63.68 GSM8K on Qwen2.5-7B).
- LoT outperforms CoT KD on pass@5 across 1–8B models in both domains, with the largest gains on OOD arithmetic (OPT-2.7B AddSub +32.11pp, SVAMP +25.09pp) and only modest in-domain gains (GSM8K +2–3pp).
- The self-evolving bandit scheduler converges faster and finishes higher than flat and staged schedules (Table 12: 42.55 vs 37.60 / 40.94 / 20.31 average).
- Moderate rewrite depth (≤3) is optimal; using the full ladder dilutes performance (Table 4: 46.09 vs 42.55 average).
Strengths
Input-side simplification is a clean, well-formalized core idea
§3.1 defines each rewrite as replacing antecedent premises A_i with their entailed conclusion c_i, so the same problem gets one inferential step precomputed per rung — distinct from LADDER-style new sub-problems; full prompts in A.8, and a 500-sample audit reports 99.2% validity, 98.0% difficulty decrease, 98.6% answer preservation.
Step-based difficulty is ablated against a real alternative
A.6 replaces step buckets with Qwen2.5-7B success-rate buckets and shows a marked drop (GSM8K 74.60→63.68; MultiArith 98.89→59.44), giving rare direct evidence for the design choice; A.5/Table 13 explain the depth effect via distribution shift (0–1-step items rise 0.1%→45.1% at depth 4+).
Broad evaluation with negative results reported, not hidden
Practically cheap pipeline with quantified overhead
Actionable depth guidance
Weaknesses
Contribution attribution favors the data, not the scheduler
No statistical testing; several in-domain claims sit within noise
Lenient, unvalidated answer verification
Cliff-edge dependence on rewriter quality, single closed model
Reproducibility gaps despite detailed appendix
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Structured Review: Ladders of Thought (LoT)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Ladders-of-Thought (LoT)?
A training recipe for 1–8B language models: a strong model (GPT-5-mini) automatically rewrites each reasoning problem into progressively easier versions by moving intermediate conclusions into the problem statement; the versions are grouped by the minimum number of remaining reasoning steps; and a multi-armed-bandit scheduler decides which difficulty bucket the student trains on at each point. The goal is to improve reasoning beyond standard chain-of-thought distillation, especially on new, easier-to-moderate problems.
How much does LoT improve small-model reasoning?
Measured as pass@5 accuracy, the largest gains are on out-of-distribution arithmetic: for OPT-2.7B, AddSub rises from 8.26 to 40.37 (+32.11 points) and SVAMP from 19.06 to 44.15 (+25.09). In-domain gains are small: GSM8K improves by about +2–3 points, and one setting (Pythia-2.8B) drops slightly (−0.45). Multi-hop results are mixed: large gains on StrategyQA and QASC, but regressions on OpenBookQA and MuSiQue for some models (up to −19.6 points for OPT-2.7B on MuSiQue).
How is LoT different from LADDER or question decomposition methods?
Decomposition methods create new sub-questions that are separate tasks. LoT keeps the same single problem and progressively replaces premise information with conclusions already derived from it, so each rewritten version is the same task made easier rather than a new sub-task — the original problem identity and final answer are preserved at every rung of the ladder.
Does LoT help larger models like 7B or 8B?
The authors trained Qwen2.5-7B and Llama3.1-8B on the same GSM8K ladder: in-domain GSM8K barely changes (+0.76 and +2.88 points), but out-of-distribution gains remain substantial (AddSub +20.18 and +31.19; SVAMP +11.03 and +10.37), with slight ASDiv regressions (−2.75 and −2.42). The main benefit at larger scales is transfer to compositional out-of-distribution tasks rather than in-domain accuracy.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–6 are not analyzable from the extracted text; the convergence and distribution claims in §4.5 and A.5 rest on the authors' figure descriptions and Table 13.
- The depth and scheduling ablations (Tables 4, 11, 12) never state the student model; their 'All'/'Self-evolving' rows numerically match OPT-1.3B's main results exactly, suggesting a single-model basis — inferred, not confirmed.
- Code and rewrite datasets are promised but unreleased (arXiv v1, 22 Sep 2026), so reproducibility statements could not be verified against artifacts.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.