Do LLMs Choose Like Humans? Using Cognitive Theory to Evaluate LLM Decision-Making
Johnathan Sun, Andrei Shleifer, Yonatan Belinkov
cs.CL · cs.LG
Quick verdict
12 LLMs show human-like context shifts in choices but not feature reweighting (only 26% human-like) on a 140k-trial benchmark; rigorous, theory-driven.
Summary
The paper tests whether LLM decision-making aligns with the cognitive theory of choice from Bordalo et al. (2026b), which explains human context sensitivity via problem categorization and attention reweighting. The authors build a 140,000-trial product choice benchmark: 100 Claude-generated products across 20 sectors, paired into 800 binary choice problems, crossed with five hedonic/functional cue types from the consumer-choice literature plus neutral trials; price and quality sweep datasets estimate feature sensitivities. Twelve open-source and commercial LLMs are evaluated at temperature zero. Results: (1) all models choose the premium option substantially more under hedonic cues (over twice as often on average), matching human choice effects; (2) only 26% of model-cue combinations show human-like reweighting of both price and quality sensitivity; (3) cue effects follow categorization signatures - in-context hedonic example share moves premium choice from 49% to 74%, and effects peak for ambiguous products and shrink with baseline certainty; (4) scale and chain-of-thought reasoning do not reliably attenuate context sensitivity. Limitations: behavioral similarity does not establish mechanistic correspondence, there is no matched human experiment on identical stimuli, and feature sensitivities are only measured indirectly via choice regressions (GPT and Gemini excluded for missing logits).
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- All 12 evaluated LLMs choose the premium option substantially more often under hedonic than functional cues (over twice as often on average), matching human context effects on choice (Section 5.1).
- Only 26% of model-cue combinations show the theory-predicted reweighting of both price and quality sensitivity, and no model does so across all cue types (Section 5.2).
- Context effects follow a categorization process: in-context hedonic evidence shifts choices from 49% to 74% premium rate, and cue effects peak for categorially ambiguous products and shrink with baseline decision certainty (Section 5.3).
- Neither model scale nor chain-of-thought reasoning reliably attenuates context sensitivity; reasoning yields human-like sensitivities only for the explicit reasoning-frame cue (Section 5.4).
Strengths
Theory-anchored predictions rather than post-hoc fitting
Section 3 derives ex-ante predictions from Bordalo et al. (2026b) - choice shift, positive beta_HR interaction in both sweeps, and three categorization signatures - avoiding the post-hoc prospect-theory fitting criticized in Section 1.
Unusually careful stimulus construction
Descriptions are length-matched (+/-5 words) and brand-free, adjacent-tier pairs use both option orders (A.1); quality is scored by an independent GPT 5.4 judge with r = 0.96 Claude-GPT agreement, a 10% manual audit, and a clean tier ladder (Table 5 Panel A: 30.4 to 89.8).
Converging categorization evidence
Strong reproducibility posture
Candid limitations section
Weaknesses
No matched human experiment
Thinking-mode comparison confounded with temperature
Feature-sensitivity analysis covers only 10 of 12 models on heterogeneous logit backends
No multiple-comparison control
Neutral-condition asymmetry under-analyzed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper tests whether LLM decision-making aligns with the cognitive theory of choice (Bordalo, Gennaioli, Lanzani and Shleifer, 2026b, QJE): context first determines problem categorization (hedonic vs functional consumption), and the category then reweights attention over price and quality. Using a 140,000-trial product choice benchmark (Table 4: 8,800 baseline + 88,000 price sweep + 44,000 quality sweep = 140,800; plus a 36,000-trial in-context arm, 176,800 total) across 12 LLMs, the authors claim: (1) human-like context effects on choice - every model picks premium more often under hedonic than functional cues, on average more than twice as often (Section 5.1); (2) non-human-like feature sensitivity - only 26% of model-cue combinations show the predicted joint reweighting of price and quality sensitivity, 12% show the opposite, 62% split (Section 5.2, Figure 2 right); (3) categorization signatures - cue effects track in-context hedonic frequency (premium rate 49% to 74% as hedonic examples go 0 to 8), peak for categorially ambiguous products, and shrink with baseline decision certainty (r = 0.13, p < 0.001; Section 5.3); (4) neither scale nor chain-of-thought reliably attenuates context sensitivity (Section 5.4). The headline conclusion: LLM decision mechanisms are distinct from human ones.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
Across 12 LLMs on a 140,000-trial product choice benchmark, hedonic context cues shift LLM choices toward premium products just as they shift human choices, but unlike humans, LLMs do not reliably reweight their sensitivity to price versus quality; larger scale and chain-of-thought reasoning do not fix this.
Which cognitive theory does the paper use, and how?
It uses the cognitive theory of choice by Bordalo, Gennaioli, Lanzani and Shleifer (2026b, Quarterly Journal of Economics): a decision-maker first categorizes a problem (hedonic vs functional consumption) based on contextual similarity to past experiences, and the category then determines attention weights on features like price and quality. The paper tests whether LLMs show the behavior this theory predicts for humans.
Which models were evaluated?
Twelve models: GPT 5.4, Gemini 3.5 Flash, GLM 5.2, DeepSeek v4 Pro, Qwen 3.7 Plus, Gemma 3 (4B/12B/27B) and Qwen 3 (4B/8B/14B/32B), plus four Qwen 3 thinking-mode configurations, for 16 configurations total. GPT and Gemini are excluded from feature-sensitivity analyses because their APIs expose no token logits.
Is the dataset and code available?
Yes. The paper states that questions, model responses, and analysis code are available at github.com/johnathansun/llm-cognitive-choice, with pinned software environments documented; data are released under CC BY 4.0 for research use.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal numeric values (per-model CI magnitudes, scatter positions) are not recoverable from text extraction; quantitative judgments rely on caption-reported statistics such as the 26%/12% quadrant shares and the 49%-to-74% in-context shift.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.