Learn Your Own Thoughts: Abstract Token Curriculum
Khashayar Gatmiry, Avrajit Ghosh, Parsa Mirtaheri, Jason D. Lee, Nika Haghtalab, Emmanuel Abbe, Peter Bartlett
cs.LG · cs.AI · cs.CL · stat.ML
Quick verdict
ATC trains latent chain-of-thought from answers alone via curriculum; strong synthetic results and a one-step attention theory, but stylized assumptions.
Summary
The paper proposes Abstract Token Curriculum (ATC), a training framework that induces continuous (latent) chain-of-thought representations from answer supervision alone, without CoT demonstrations or hand-designed scratchpads. ATC organizes training as a sequence of increasingly difficult input distributions; after each phase saturates, the model's last hidden embedding is appended to the context as a latent thought token. Because harder stages can corrupt earlier representations, the authors add curriculum backtracking (return to stages whose accuracy regressed), truncated backpropagation (gradients through a fixed window of recent thoughts), and candidate dropout (hiding answer candidates on 75% of examples to block copying shortcuts). Theoretically, for parity with single-layer softmax attention, one sufficiently large gradient step on the keys concentrates at least 99% of answer attention on the most recent CoT token with probability at least 19/64, instantiating the 'attention simplicity bias' conjectured by Abbe et al. (2024). Empirically, on 10-of-20-bit parity ATC stays near-perfect where a data-curriculum-only baseline degrades; on shortcut-controlled graph reachability (2-layer GPT-2, 15M parameters) it matches Coconut's 99.9% accuracy, reaching 95% in 51 epochs versus 132-468 for Coconut variants, without intermediate supervision; truncated backprop with backtracking retains 97.4% at 0.27x memory. Eight-digit addition reaches 98.7-99.0% exact match. Main limitations: the theory assumes correlation-1 prior thoughts and keys-only updates, and all tasks are synthetic with hand-designed difficulty orderings.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ATC elicits effective continuous chain-of-thought representations using only answer supervision, by scheduling increasingly difficult input distributions and recycling saturated hidden states as latent thought tokens.
- For parity with one-layer softmax attention, a single sufficiently large gradient step on the key parameters concentrates at least 99% of answer attention on the most recent CoT token with probability at least 19/64, establishing the attention simplicity bias conjectured by Abbe et al. (2024) in this setting.
- On shortcut-controlled graph reachability, ATC matches Coconut's 99.9% test accuracy without intermediate supervision and reaches 95% accuracy in 51 epochs versus 132-468 epochs for Coconut variants (Table 1).
- Curriculum backtracking is necessary under truncated backpropagation: with backward window b=5, advance-only training collapses to 51.0% while backtracking retains 97.4% at 0.27x peak memory (Table 2).
- On eight-digit addition with carry, ATC reaches 98.7-99.0% exact match while Coconut stalls at 0% and removing candidate dropout (ρ=0) leaves ATC stuck at stage 0 (Table 3).
Strengths
Well-controlled empirical design
Table 1 includes four controls that cleanly decompose the contribution: No-CoT 48.3%, discrete CoT 63.5%, full-CoT-no-curriculum 47.8%, No-CoT+data-curriculum 49.0% — all fail where ATC reaches 99.9%, isolating the interaction of curriculum and latent compute.
Genuinely supervision-free latent CoT
ATC requires neither discrete CoT traces (as Coconut does) nor intermediate-state labels (as Zhu et al. 2026b); the only supervised token is the final answer, and curriculum gating uses task accuracy — a clean conceptual contribution (Sec. 6.2).
Theory-experiment loop closes
Practical training-cost result with honest ablations
Useful sensitivity analysis
Weaknesses
Stylized theory with weak success guarantee
Synthetic-only evaluation with hand-designed difficulty axis
Thin statistical reporting
Hyperparameter fragility only partially characterized
No code or artifact release statement
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a concrete gap in the latent chain-of-thought literature: continuous-CoT methods (Coconut, Hao et al. 2025; Zhu et al. 2026b) train latent thoughts only under supervision of intermediate states or pre-existing CoT demonstrations. ATC claims that answer supervision alone suffices if training follows a difficulty curriculum. Three claim types are advanced: (1) methodological — recycling saturated hidden states as latent tokens across curriculum phases elicits useful continuous thoughts (Sec. 2); (2) theoretical — one gradient step on attention keys concentrates answer attention ≥99% on the most recent thought with probability ≥19/64 (Theorem 1), instantiating the 'attention simplicity bias' conjectured by Abbe et al. (2024); (3) empirical — parity (Sec. 5, Fig. 2), shortcut-controlled graph reachability (Sec. 6, Table 1) and eight-digit addition with carry (Sec. 7, Table 3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Abstract Token Curriculum (ATC)?
ATC is a training method that teaches a transformer to form internal, continuous 'chain-of-thought' representations using only the final answer as supervision. It gradually increases task difficulty across curriculum stages and, after each stage is learned, feeds the model's own last hidden state back into the context as a latent thought token for harder instances.
Does ATC need chain-of-thought data or hand-designed scratchpads?
No. Unlike Coconut and other latent-CoT methods, ATC never supervises intermediate reasoning states and needs no CoT demonstrations; its only training label is the final answer, and curriculum progression is gated on answer accuracy on a validation set.
What results does the paper report?
On 10-of-20-bit parity, ATC stays near-perfect while a data-curriculum-only baseline degrades. On shortcut-controlled graph reachability with a 15M-parameter GPT-2, ATC reaches 99.9% test accuracy and hits 95% at epoch 51, versus epochs 132-468 for Coconut variants that need intermediate supervision. On eight-digit addition with carry it reaches 98.7-99.0% exact match, where Coconut stalls at 0%.
What are the main limitations of the paper?
The theorem covers only a single gradient step on attention keys under an idealized correlation-1 assumption, with about a 30% success probability; all experiments are small synthetic tasks whose difficulty orderings are designed by hand; and statistical reporting (seed counts, confidence intervals) is thin for some headline numbers. No code release is mentioned.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2, 4-7 could not be analyzed visually; only their captions and in-text numerical descriptions were available.
- The extracted text ends within Appendix H.4; any content after that point (further appendices, code/data statements) may be missing, so 'codeAvailable: false' reflects absence in the provided text.
- Theorem proofs were reviewed at the level of statements and proof structure; line-by-line verification of the perturbation algebra in Appendices C-G was not performed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.