Large Language Models Develop Belief State Geometry In-Context
Daniel Balcells, Andrew Jun Lee, Chirag Rastogi, Paul M. Riechers, Adam Shai, Xavier Poncini
cs.LG · cs.CL
Quick verdict
Evidence that six LLMs linearly encode and causally use HMM belief states in-context is well-controlled but confined to four author-designed HMM families.
Summary
The paper tests whether LLMs doing in-context learning on hidden-Markov-model (HMM) token streams internally represent the belief state — the posterior over hidden states — that computational mechanics predicts optimal prediction requires. Six open-source LLMs (Qwen3.5-9B/4B, Llama-3.1-8B, Llama-3.2-3B, Gemma-4-E4B/E2B) are prompted with 20,000-token sequences from 40 HMMs (four families, 10 parametrizations each; 30 newly designed). Behaviorally, converged KL to ground-truth next-token distributions is 0.009-0.018 nats for main-text parametrizations, matching order-4.2-11 Markov approximations (median crossover k 3.4-4.7 across all 40). Linear probes decode beliefs from the residual stream with peak held-out R² 0.83-0.99, far above shuffled/random-belief controls; transfer-probe matrices show parametrization-specific information; belief decodability exceeds what NTP, log-NTP, or order-one structure explain, tracking theoretical surplus (Pearson r 0.85-1.00; Arch log-NTP 0.43-0.88). Causally, steering the probe-identified subspace installs target log-NTP decodability downstream, NTP-preserving δ-direction steering succeeds where magnitude-matched random directions differ, and patching shifts predictions toward injected beliefs while random-belief controls fail; self-repair appears at early layers. Main limitations: only four low-dimensional author-designed HMM families in a slow-mixing region, steering results exclude the Gemma class, and interventions reuse the probes' regression embedding.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Belief states of 40 HMMs are linearly decodable from the residual stream of six open-source LLMs, with peak held-out probe R² of 0.83-0.99, far above shuffled and random-belief controls.
- The decodable information is specific to the HMM parametrization and exceeds what NTP, log-NTP, or order-one suffix structure can linearly explain, tracking the theoretically derived belief surplus.
- Steering the probe-identified belief subspace installs the target's implied log-NTP downstream and an NTP-preserving δ-direction component, while magnitude-matched random directions behave differently.
- Patching the belief subspace shifts final-token predictions toward injected target beliefs even when targets conflict with the un-intervened prefix, with random-belief injections failing.
- Together these results extend the computational-mechanics belief-state-geometry account from toy transformers trained on HMMs to production-scale open-source LLMs.
Strengths
Falsifiable, theory-first design
Unlike post-hoc ICL representation studies, the target geometry (the belief simplex) is derived from computational mechanics (Shalizi & Crutchfield 2001; Section 1-2) before any probing, making the hypothesis precise and rejectable.
Layered control battery
Shuffled and random-belief probes (Fig 3), cross-parametrization transfer probes correlated against ground-truth R² matrices (Fig 4), order-k baselines (Figs 2, 5), NTP-preserving δ∈Ker(Mᵀ) steering vs magnitude-matched random direction (Fig 6), and random-belief patching control (Fig 7) — each claim has a matched null.
Breadth with flagged exceptions
Converging causal chain
Honest calibration of confounds
Weaknesses
Narrow process class
Probe-guided intervention circularity
Thin statistical treatment
Universality exceptions underdiscussed at claim level
Bayesian gloss outruns the behavioral analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Large Language Models Develop Belief State Geometry In-Context
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a 'belief state' in this paper?
The posterior probability distribution over a hidden Markov model's hidden states, given the token history observed so far. Computational mechanics argues that optimally predicting the next token of HMM-generated data requires tracking this quantity, and as it evolves it traces a characteristic low-dimensional geometry (roughly a simplex) in representation space. The paper tests whether LLM activations contain exactly that geometry.
Do LLMs really perform Bayesian inference in context, according to this paper?
The authors provide representational evidence (linear decodability of belief states with R² up to 0.99) and causal evidence (steering/patching the identified subspace shifts predictions toward injected beliefs) that six open-source LLMs approximate optimal Bayesian prediction when prompted with sequences from four families of small hidden Markov models. The claim is specific to this controlled setting — it is not demonstrated for natural-language tasks, and the smallest model tested (Gemma-4-E2B) shows weaker effects.
Which models and how many HMMs were tested?
Six open-weight LLMs — Qwen3.5-9B and Qwen3.5-4B, Llama-3.1-8B, Llama-3.2-3B, and Gemma-4-E4B and Gemma-4-E2B — were prompted with 20,000-token sequences from 40 HMMs (Mess3, Arch, Wing, and Strata families, 10 parametrizations each; 30 of the processes newly designed), with 10 sequences per HMM.
Is the code or data for this paper available?
The extracted text does not mention a code or data release. However, the transition matrices of all four HMM families are fully specified in Appendix C, and the probe, steering, patching, and tuned-lens procedures are described in enough detail that the synthetic-data pipeline and analysis can be reimplemented independently using open-weight models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Parts of Section 5.2 (order-k analysis), Section 9 (Discussion), and Appendices E, P, Q were truncated in text extraction; judgments on those parts rely on cross-references in visible text.
- Figures are available only as extracted captions/tick labels; numeric readings of curves and heatmaps cannot be verified visually.
- No code/data availability statement appears in the extracted text; absence of a release is inferred, not confirmed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.