Beliefs and Behavior in Language Models
cs.AI · cs.LG
Quick verdict
Latent-variable test of whether LLMs hold beliefs; rigorous, convincing for capable models, with survivorship-bias and bottleneck-ablation caveats.
Summary
The paper asks whether 'belief' is a predictively useful abstraction for large language models. The authors fit a latent-variable model in which a scalar degree of belief z, inferred from an LLM's responses across sixteen prompts per case (numeric probability, Likert and quintile judgments, plain decisions, and threshold-, bet-, and cost-framed decisions), predicts held-out outputs in three domains: chronic-kidney-disease screening on NHANES data, Werewolf games, and forecasting of social-science experiments, across nine models from Gemma 3 4B to GPT-5.6 Terra. For the most capable models a single latent reconstructs behavior with concordance 0.8–0.95 and out-of-sample R² of 60–90%, generalizing zero-shot across decision framings and domains (AUC near 1, pooled R² 0.6–0.8); verbalized probabilities alone capture over 90% of this predictive performance. Weaker models show monotone but noisy latent structure, systematic framing effects, and decision thresholds far from instructed values. Branching experiments on two models find roughly half the variance in elicited probabilities attributable to the sampled reasoning prefix (ICC ≈ 0.5), supporting instance-level belief attribution. Main limitations: strict complete-case parsing drops up to 37% of cases for the weakest model, with no dropped-case analysis; the one-dimensional latent bottleneck is never compared against a direct prediction baseline; and the capability-gradient claim rests on visual ordering by a third-party index.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A single scalar latent belief inferred from 16 outputs per case predicts held-out LLM actions with concordance 0.8–0.95 and out-of-sample R² of 60–90% for the most capable models across three domains.
- Behavioral coherence and predictability from an inferred belief track the capability gradient across nine models from Gemma 3 4B to GPT-5.6 Terra.
- Verbalized numerical probabilities are high-quality belief measurements, capturing over 90% of the joint latent-inference predictive performance for the best models.
- Outward incoherence in weaker models (framing effects, miscalibrated thresholds) is consistent with a shared latent belief decoded on a non-instruction-following scale rather than with absent beliefs.
- Beliefs are partly instance-specific: roughly half of within-case variance in elicited probabilities is attributable to the sampled reasoning prefix (ICC ≈ 0.5).
Strengths
Decision-theoretic operationalization of a fuzzy question
Grounds the question in formal properties — sufficiency and cyclic monotonicity from Yamin et al. (2026) — enforced via monotone Beta/logit/ordered-logit observation models, and evaluates elicited probabilities as measurements rather than equating them with belief (Fig. 4 sufficiency analysis).
Matched-triplet framing design
Threshold t = 1/(r+1), r:1 bet, and r:1 cost ratio for r ∈ {2,4,9,19} create payoff-identical prompts, cleanly separating instruction-given threshold compliance from framing sensitivity (Fig. 5); bet-framed conservatism in weak models is a concrete, interpretable finding.
Negative-control expressiveness checks
Zero-shot generalization as anti-overfitting evidence
Unusual transparency
Weaknesses
Survivorship from strict complete-case parsing
The latent bottleneck itself is never ablated
Capability-gradient claim lacks formal statistics
Instance-level evidence covers 2 of 9 models
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do LLMs have beliefs, according to this paper?
The paper takes a behavioral stance: for highly capable models, a single inferred latent belief predicts held-out actions across 16 prompt types with concordance 0.8–0.95 and out-of-sample R² of 60–90%, so treating them as holding degrees of belief is predictively useful. The authors explicitly do not claim to settle whether LLMs 'really' have beliefs in a philosophical sense.
How were the beliefs measured?
Each of 500 cases per domain is posed to a model through 16 prompts — numerical probability, Likert scale, quintile bins, a plain decision, and parameterized decisions using probability thresholds, bets, and asymmetric error costs. A latent-variable model with a fitted prior infers one scalar belief per case; predicting held-out outputs from a subset of the responses quantifies belief quality.
Which models were tested, and did weaker models fail?
Nine: Gemma 3 4B/12B/27B, Llama 3.1 70B, Qwen3 32B, DeepSeek V3.2, GLM-5, and GPT-5.6 Luna/Terra. Weaker models still showed monotone latent structure, but with much lower predictive fidelity, systematic framing effects, and decision thresholds far from instructed values; performance generally tracked the capability ranking.
What data and domains were used?
Three: chronic-kidney-disease screening on US NHANES patient profiles, Werewolf games from the Kaggle Agent Arena dataset (LLM-vs-LLM transcripts), and forecasting the outcomes of social-science survey experiments from the Ashokkumar et al. (2026) dataset.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are image-only in the extraction; per-model point values were taken from caption text and ranges stated in prose, not read off plotted data.
- Extraction is complete through Appendix E.3.7 (the end of the prompt catalog); no truncation observed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.