Before You Poll with LLMs: A Deliberative Diagnostic Framework
cs.CL · cs.AI · cs.CY
Quick verdict
Five frontier LLMs all misrepresent how personas revise beliefs after balanced information, but the human-LLM delta gap confounds a text briefing with A1R's richer deliberation.
Summary
This paper introduces the Deliberative Polling Diagnostic Framework, a protocol for testing whether LLM personas update beliefs the way humans do after exposure to identical balanced information — a capacity the authors call dynamic fidelity, which existing static evaluations of silicon sampling never measure. The four-phase design (pre-poll, briefing, post-poll, comparison of Delta = Post - Pre) is applied to 526 personas and 72 questions from America in One Room (2019, NORC address-based probability sample) and to five instruction-tuned frontier models queried at temperature 0 on a 0-10 integer scale: GPT-5.1 at full scale (526x72), Gemini 2.0 Flash at 526x20, and Claude Sonnet 4.5, Llama 3.3 70B and DeepSeek V3 on 100-persona subsamples. No model reproduces the human outgroup shift of Delta = -0.21. GPT-5.1 reverses (+0.50, d = 0.27); Gemini (-1.36), Claude (-1.04) and Llama (-1.42) overshoot at roughly 5-7x human magnitude; DeepSeek is rigid (+0.02, p = 0.051). Ablations show failures are content-triggered (irrelevant Wikipedia briefings collapse both GPT-5.1 and Gemini to -0.75) and identity-specific (GPT-5.1 reverses on outgroup, +0.24, but overshoots on ingroup, -1.09; Gemini shows the inverse), which the authors name self-sycophancy. Key limits: one US poll from 2019, models receive only the informational channel of a much richer human treatment, and the evidence is behavioural only.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No existing benchmark tests whether LLM personas revise beliefs as humans do after new information; the Deliberative Polling Diagnostic Framework fills that gap by comparing human and LLM belief shifts after identical balanced briefings.
- All five frontier models fail the diagnostic, each in a distinct way: GPT-5.1 reverses (outgroup Δ = +0.50 vs human −0.21), Gemini 2.0 Flash, Claude Sonnet 4.5 and Llama 3.3 70B overshoot at 5–7× human magnitude, and DeepSeek V3 is rigid (Δ = +0.02).
- The failures are triggered by policy content rather than being architectural: replacing balanced policy briefings with irrelevant Wikipedia text collapses both GPT-5.1 and Gemini to an identical Δ = −0.75.
- The failures are identity-specific and content-selective: the same model and persona reverse on outgroup questions (+0.24) but overshoot on ingroup questions (−1.09), and 80% of outgroup questions reverse versus 26% of policy questions.
- These patterns constitute 'self-sycophancy' — conformity to the model's internal stereotype of the assigned persona rather than reasoning from the briefing — and no tested intervention fixes direction and magnitude simultaneously.
Strengths
A genuinely new evaluation axis with a clean operationalisation
The paper is right that every prior silicon-sampling benchmark tests static fidelity (Santurkar 2023, Bisbee 2024, Chuang 2024b, Kang 2025 all cited in §2.1). Turning deliberative polling into a pre/post diagnostic and defining Δ = Post − Pre as the unit of comparison (§3.1) gives a directional, falsifiable target: humans moderate on outgroup hostility (Δ = −0.21, Table 1), so a model that moves the other way is unambiguously wrong rather than merely miscalibrated.
Mechanism experiments that actually discriminate between hypotheses
§6 runs three ablations that bite. The content-trigger swap (irrelevant Wikipedia text on pasta, chess, gardening; 526 personas, 20 questions) collapses GPT-5.1 and Gemini to the identical Δ = −0.75 (Table 4), which rules out a general architectural inability to moderate. The ingroup/outgroup control (§6.2) makes the same model, persona and briefing produce opposite failure modes purely from the identity target (GPT-5.1: +0.24 outgroup vs −1.09 ingroup, t = 12.29, d = 0.43; Gemini: −0.34 vs +0.42, t = −12.60, d = −0.44), which is far stronger evidence than any single aggregate gap.
Honest statistical housekeeping
Unusually thorough appendix documentation
The intervention analysis is deliberately framed as exploratory and still yields the most useful practical result
Weaknesses
The human and LLM treatments are not equivalent, and this is the load-bearing comparison
Pre-poll means already diverge sharply, which threatens the Δ = Post − Pre defence
The outgroup category is five items, and the selectivity test fails the paper's own correction
Scale is not matched across models but is presented as if it were
'Self-sycophancy' is an interpretation, not a measured mechanism
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a gap that is genuinely empty: every prior evaluation of silicon sampling (Argyle et al. 2023; Santurkar et al. 2023; Bisbee et al. 2024; Chuang et al. 2024b; Kang et al. 2025) asks whether an LLM persona holds the *right opinion* at a single moment. This paper asks whether the persona *updates* that opinion the way a human does, which it calls dynamic fidelity. It contributes three things (§1): the Deliberative Polling Diagnostic Framework itself, evidence that all five tested frontier models fail it in distinct ways, and a behavioural characterisation of a failure family it names self-sycophancy.
Two research questions structure the work. RQ1 (§5) asks whether and how LLM belief revision diverges from humans, by model, question type and partisan identity. RQ2 (§6) asks what mechanism drives the failures. The scope is deliberately narrow on three axes: one deliberative poll (America in One Room, 2019, US national politics), instruction-tuned models only, and the informational channel only. Each of those restrictions is declared, and each turns out to matter for how far the results travel.
The headline claim — that no model matches human belief revision and that each fails differently — is well supported within that scope. The stronger claim, that the failures reveal a declarative-procedural gap in LLM cognition and a self-reinforcing stereotyping loop (§8.3), rests on inference from behaviour and is appropriately hedged by the authors.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Deliberative Polling Diagnostic Framework and what does it test?
It is a four-step protocol for checking whether LLM personas change their opinions the way real people do. You poll personas before and after giving them the same balanced briefing materials that a human deliberative poll used, then compare the shift (post minus pre) for humans and models. Unlike standard checks, which only ask whether a persona holds the right opinion at one moment, this tests whether it updates that opinion correctly. Behaviour is then classified as match, overshoot (right direction, too large), reversal (wrong direction) or rigidity (no change).
Do large language models update political opinions like humans after balanced information?
No. In the study, humans became slightly less hostile toward the opposing party after deliberation (a shift of −0.21 on a 0–10 scale), while GPT-5.1 personas became more hostile (+0.50). Gemini 2.0 Flash (−1.36), Claude Sonnet 4.5 (−1.04) and Llama 3.3 70B (−1.42) moved in the right direction but 5 to 7 times further than humans, and DeepSeek V3 barely moved (+0.02). The pattern was consistent across Democrats, Republicans and Independents.
What is self-sycophancy in LLMs?
Self-sycophancy is the tendency of a model to answer in line with its own internal stereotype of the persona it was assigned, instead of reasoning from the information it was just given. It differs from ordinary sycophancy because the model is not deferring to a user; it is deferring to a cached image of, say, how a partisan is supposed to feel about the other side. In this study GPT-5.1 conformed to a stereotype of outgroup hostility, while Gemini conformed to a stereotype of ingroup favouritism.
Can prompting fixes such as chain-of-thought or removing party labels stop this behaviour?
Not completely. In the paper's exploratory tests, multi-agent deliberation cut GPT-5.1's reversal rate from 80% to 20% but made Gemini's overshoot worse (6.5 to 18 times human size). Chain-of-thought and putting the persona description first removed Gemini's reversal but left about 5.7× overshoot. Removing party labels gave the most balanced improvement (reversal 40% to 20%, overshoot 6.5× to 4.3×), but the model still moved far more than humans. No single intervention fixed both direction and magnitude.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1, 2 and the item-level charts are not analysable from the extracted text; all numeric claims come from Tables 1–15 and the body text.
- Appendix C demographic stratification (Table 13) and label-swap (Table 12) results are summarised in the text rather than reproduced in full, so effect sizes there are read at the level the authors report.
- The paper evaluates proprietary API endpoints (OpenAI, Google, Anthropic, DeepInfra); the extracted text cannot verify provider-side moderation or post-processing effects that Limitation 10 raises.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.