How Many Thoughts Can a Vector Hold? The Capacity of Reasoning by Superposition
Hongyu Gu, Chang Liu, Jingwen Fu
cs.AI
Quick verdict
Proves cumulative superposition needs fewer dimensions than frontier for nonlocalized queries, backed by small-scale experiments; solid theory, narrow scope.
Summary
The paper asks what a continuous latent thought should preserve during multi-step reasoning: only the current frontier of alternatives, or the full history of reached states. Under Gaussian random embeddings with a nearest-candidate linear decoder, the authors derive an exact score-competition law and a sharp phase transition: readout succeeds once the alignment μ satisfies dμ² ≳ 2 log M for M competing candidates. For C-nonlocalized query classes drawing on most of the reached set, they prove cumulative superposition needs at most C⁴k/m times the dimension of frontier superposition — vanishing when the frontier grows slower than the history (k = o(m)). A two-attention-block Transformer with a residual and RMSNorm implements both updates, and the smallest retained coefficient is proven to be the routing bottleneck; when future use is unknown, uniform cumulative weighting is exactly minimax-optimal. Experiments on ProsQA graph-4 traces confirm the separation: a trained two-block router crosses 90% accuracy at width 40.3±1.1 (cumulative) versus 110.6±1.6 (frontier), and a two-layer GPT-2 replication gives 34.4±3.8 versus 66.2±5.9. Weighted-routing ablations show uniform weights reach 0.906 joint route success versus 0.586 for recency-heavy weighting. Limitations: the theory assumes fixed Gaussian codebooks and linear decoding, experiments use ~23-node graphs with five seeds, and the reversal applies only to nonlocalized queries — frontier remains better for single-object readouts.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- For C-nonlocalized query classes, cumulative superposition needs dimension at most C⁴k/m(1+o(1)) times frontier superposition, a ratio that vanishes in the growing-history regime k=o(m) (Theorem 5.1).
- Reliable superposition readout obeys a sharp phase transition at dμ²/(2log M)=1 with critical dimension d⋆=(2+o(1))·2log M/μ² (Theorem 4.2, Corollary 4.4).
- A two-attention-block residual Transformer with RMSNorm realizes both frontier and cumulative updates without FFNs, and the smallest retained coefficient governs routing width (Theorem 5.3).
- Uniform cumulative weighting is exactly minimax-optimal over the full nonlocalized query family and within the sharp factor C² for any fixed nonlocalized class (Theorem 5.5).
- Paired five-seed experiments on ProsQA graph-4 confirm cumulative reaches 90% held-out accuracy at roughly 0.37–0.52× the frontier critical width in both a two-block router and a two-layer GPT-2.
Strengths
Exact, checkable dimension theory
Lemma 4.1 gives an exact Gaussian score decomposition (signal μχ_d vs. M i.i.d. normals); Theorem 4.2 proves a genuine zero-one phase transition at dμ²=2log M and Corollary 4.4 a matching law d⋆=(2+o(1))·2log M/μ², with complete proofs in Appendix A rather than heuristic scaling arguments.
Sharp, proven reversal with matching witness
Theorem 5.1 bounds the cumulative/frontier dimension ratio by C⁴k/m for all C-nonlocalized query classes, and tightness is established via an explicit witness query uniform on ⌈m/C²⌉ coordinates containing F (Appendix B), so the headline result is an equality-case theorem, not just an inequality.
Theory-to-architecture bridge
Quantitatively consistent, well-controlled experiments
Actionable minimax design rule
Weaknesses
Large theory-practice representational gap
Tiny empirical scale
Reversal is query-class-dependent and the favorable-opposite regime is never shown
Unexplained quantitative theory-experiment gap
Appendix duplication
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses a design question in latent (continuous) reasoning — COCONUT-style continuous thoughts, looped Transformers, and reasoning by superposition (Zhu et al., 2025b): should a fixed-dimensional thought keep only the current frontier of alternatives or the full history of reached states? The authors formalize frontier (uniform on F, |F|=k), cumulative (uniform on R, |R|=m), and general weighted superpositions over a directed graph, decoded by top-1 inner product against a fixed bank of M decoy directions. Three linked claims: (1) an exact dimension law — readout succeeds once dμ² exceeds ≈2 log M (Theorem 4.2, Corollary 4.4); (2) for C-nonlocalized query classes (‖u‖∞ ≤ C/√m), cumulative needs at most C⁴k/m times frontier's dimension, vanishing when k=o(m) (Theorem 5.1); (3) uniform cumulative weighting is exactly minimax-optimal when future use is unknown and within the sharp factor C² for any fixed class (Theorem 5.5). The claims are carefully scoped in the theorems, but the abstract's framing invites broader reading than the nonlocalized scope supports.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
That retaining the full reasoning history in a continuous latent state (cumulative superposition) can require fewer hidden dimensions than keeping only the current alternatives (frontier superposition), because historical components add coherently for queries that aggregate over history. For nonlocalized queries the required dimension ratio is bounded by C⁴k/m, which vanishes as history outgrows the frontier.
Was the claim tested on real large language models?
Partially. Besides analytic Monte Carlo simulations, the authors trained a lightweight two-attention-block router and a two-layer GPT-2 on ProsQA graph-4 traces (graphs of ~23 vertices, solution length ~3.5). Cumulative crossed 90% held-out accuracy at roughly half or less the width needed by frontier. No full-scale pretrained latent-reasoning model was tested.
Does this mean models should always keep full reasoning history?
No. The advantage is proven for nonlocalized queries that reuse most of the reached history. For localized readouts of a single object, a frontier state has larger alignment (1/√k versus 1/√m) and is more dimension-efficient. Uniform cumulative weighting is minimax-optimal only when future use is unknown and exchangeable.
Is code available for this paper?
The paper does not state a code release. However, Appendix D provides complete configurations (widths, seeds, optimizer settings, loss weights), uses the public ProsQA graph-4 splits released with Zhu et al. (2025b), and reports modest total compute (~38 GPU-hours on A100s for the main sweep), making reimplementation feasible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels (Figs. 2-4) were extracted as jumbled axis/legend fragments; panel-level readings in this review rely on captions and the Appendix D tables rather than the plotted curves.
- Inline mathematics is flattened by PDF extraction (e.g., subscripts/superscripts run together); proofs were assessed at the level of the reconstructed argument lines, not symbol-by-symbol.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.