Auditing Political Alignment in LLM Assistants: Engagement, Stance, and User Identity
cs.CL · cs.AI · cs.CY
Quick verdict
Claims LLM political speech follows user-conditional 'speech regimes'; a preregistered 7,500-conversation audit is persuasive but rests on LLM judges.
Summary
This paper argues that an AI assistant's political behavior is not a fixed ideological position but a 'speech regime': a policy over whether to engage, which stance to take, and what a refusal transmits, conditional on the topic and what the system infers about the user. Two dimensions yield a five-type typology (engager, abstainer, selective abstainer, conditional engager, fixed position). In a preregistered correspondence-style audit, scripted users with randomly assigned political identities held 7,500 multi-turn conversations with six systems (GPT-5.5, Claude Opus 4.8, Grok 4.3, Gemma 4, Mistral Small 3.2, DeepSeek V4) across five topics: abortion, Catalan independence, climate change, Nazism, and a pineapple-on-pizza control. Two LLM judges scored 27,828 answers against human-coded validation, with refusal treated as an outcome. Every system accommodates users on the control, showing political restraint is a policy. On abortion, GPT refuses only 7% and mirrors users, Claude answers strongly conservative users 35% of the time versus roughly 5% for everyone else, and Gemma refuses all 900 answers. Five systems hold firm on climate change and Nazism; only Grok accommodates climate skeptics and mirrors abortion identity on gun control. A Grok 3 versus 4.3 comparison shows the regime shifting between versions while averages stay flat. Key limitations: the LLM judges include two studied systems, human validation rests on the author's own pilot coding, and some findings depend on small cells.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 4
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An AI assistant's political behavior is a conditional policy — a 'speech regime' over engagement, stance, and transmission — not a fixed ideological position, and it varies by user, topic, and release.
- All six systems accommodate users on the zero-stakes control topic, showing that political restraint on contested topics is a policy choice rather than a missing capability.
- On abortion the systems occupy different regimes: GPT engages and mirrors everyone (7% refusals), Gemma refuses all 900 answers, Claude answers strongly conservative users 35% of the time versus ≤5% for others, and Grok accommodates conservatives only.
- Developer rules constrain the direct answer more than engagement or transmission, so accommodation can spill over to unmentioned topics — only Grok mirrors abortion identity on gun control.
- Speech regimes change across releases (Grok 3 → 4.3) while average stance stays flat, so average-based bias audits miss regime change.
Strengths
Preregistered, randomized-user design
Registration (Zenodo DOI 10.5281/zenodo.21135155, July 2, 2026) fixed systems, scripts, scoring instruments, cell sizes, and estimands (e.g., neutral range [4,6] on abortion/Catalan; target 10 on climate/Nazism) before collection; correspondence-study logic applied to multi-turn LLM conversations is a genuine design advance over single-turn prompt-slant audits.
Refusal treated as an outcome, not missing data
This uncovers patterns average-bias scores miss: on abortion Claude answers 48/139 strongly conservative users (35%) while refusing 97% of everyone else, and Gemma refuses all 900 answers; §7 shows several systems look 'centrist' only when refusals are scored 5.
Unusually extensive robustness battery
Manipulation and mechanism checks
Release comparison with a sharp negative result
Weaknesses
LLM-judge circularity
Thin, non-independent human validation
Small-cell fragility in secondary claims
Post-hoc scoring adjustment on a headline topic
Ecological and definitional scope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper reconceptualizes the politics of consumer AI assistants. Its central claim is that an AI system's political behavior is not a fixed ideological position but a **speech regime** — a policy π(e, s, τ | u, t) over engagement (whether to answer), stance (the position taken), and transmission (what a refusal or tangent conveys), conditional on topic t and the user identity u the system infers (§3). From these dimensions it derives five ideal types (engager, abstainer, selective abstainer, conditional engager, fixed position) and a developer-cost theory (§4) yielding expectations E1–E5. Empirical claims: regimes diverge most on contested topics; all systems accommodate on a zero-stakes control, so political restraint is a policy; anchored topics hold except Grok on climate; accommodation can spill to unmentioned topics (Grok on gun control, §6.5); and regimes change across releases while averages stay flat (§6.6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a 'speech regime' in this paper?
It is the policy an AI assistant follows for political questions: whether it engages at all, which stance it takes when it answers, and what a refusal conveys — conditional on the topic and what the system infers about the user. The paper derives five regime types: engager, abstainer, selective abstainer, conditional engager, and fixed position.
Which AI systems were audited and what was found?
Six: GPT-5.5 (OpenAI), Claude Opus 4.8 (Anthropic), Grok 4.3 (xAI), Gemma 4 (Google), Mistral Small 3.2, and DeepSeek V4. All systems accommodated users on a harmless control topic (pineapple on pizza). On abortion, most refused most users: GPT refused 7% and mirrored users, Claude answered strongly conservative users 35% of the time versus about 5% for everyone else, and Gemma refused all 900 answers.
Is the study preregistered and are the data public?
Yes. It was preregistered on Zenodo (DOI 10.5281/zenodo.21135155) on July 2, 2026, with two dated amendments made before analysis. The paper states that transcripts, judge labels, rubrics, persona scripts, and analysis code will be deposited in a public archive at journal publication.
What are the main limitations of the study?
Scoring relies on two LLM judges that are themselves among the audited systems; human validation is limited to the author's own coding of a 700-answer pilot; findings cover six systems, five topics, one language (English), and one time window, and some secondary results rest on small numbers of answered questions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text omits several sections and appendices (parts of §6.1, Appendices I–N), including the tables behind the Grok 3 vs 4.3 release comparison; those findings are assessed from the main-text narrative only.
- Figures (1–6, C.1, D.1, H.1) could not be analyzed; quantitative claims rely on tables and text present in the extraction.
- Some appendix material (e.g., vaccine-safety Appendix L detail, serving configuration for local models) is absent from the extraction; the review reflects only the provided text.
- Model names and dates are as printed in the preprint (e.g., 'GPT-5.5', 'Claude Opus 4.8', 'Grok 4.3') and could not be externally verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.