Do LLMs Trust the Accuser or the Accusation? Measuring Belief Shifts in Werewolf
Yu-Yu Yang, Ti-Rong Wu, Hung Guei, Hsing-Yu Chen, I-Chen Wu
cs.AI · cs.CL
Quick verdict
Belief-shift benchmark finds 40 open-weight LLMs (≤120B) accept accusations from distrusted or wolf-aligned sources; self-report and anchoring caveats apply.
Summary
The paper introduces a belief-shift benchmark that measures how accusation messages change the stated beliefs of large language models observing Werewolf games. The authors generate 200 seven-player games through LLM self-play, use Claude Opus 4.6 to annotate 1,224 suspicion and accusation messages (verified at 96% agreement against two human annotators on a 150-message subset), and ask observing village-side models to report prior beliefs about the accuser and target on a seven-level scale, then report belief shifts after seeing the message. Evaluating 40 open-weight configurations (29 checkpoints, 1B–120B), they find models separate true wolves from villagers increasingly well with scale (Spearman ρ=0.54), yet direct accusations still shift suspicion toward the target (+0.98 for village-side accusers, +0.74 for wolf accusers) and reduce suspicion of the accuser. Trusted accusers sway models regardless of their true team (target shift 0.94 vs 0.96), while larger models better resist distrusted accusers (ρ=−0.37). A preliminary frontier-model experiment shows this credulity pattern partially reverses at the frontier. Main limitations: beliefs are self-reported with the prior fed back (anchoring risk, acknowledged but untested), all dialogue is LLM-generated within one game environment, and multiple rank correlations are reported without multiple-comparison correction.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Open-weight LLMs (1B–120B) discriminate true wolves from villagers in prior beliefs, and this ability improves with scale (separation 0.80 in ≤4B to ~1.3 above 10B, Spearman ρ=0.54, p=4×10⁻⁴).
- Direct accusations shift observer suspicion toward the accused target (avg +0.98 for village-side accusers, +0.74 for wolf accusers) and reduce suspicion of the accuser, with limited sensitivity to the accuser's true team.
- Prior trust dominates accusation content: trusted accusers produce nearly identical target shifts whether village-aligned (0.94) or wolf-aligned (0.96), while distrusted accusers still produce smaller positive shifts (0.33 / 0.14).
- Larger open-weight models make better use of prior distrust, with wolf-accuser target shifts falling from 0.30 (≤4B) to −0.12 (≥30B) in the Distrust bin (ρ=−0.37, p=0.018).
- Six frontier models partially break the credulity pattern, discounting accusations from wolf-side accusers (target shifts −0.10/−0.13) and becoming more suspicious of the accusers (+0.53/+0.50).
Strengths
Novel process-level metric beyond win/loss
Shifts evaluation from final game outcomes to per-message belief updating (§1, §3), directly instrumenting the source-versus-content trade-off that outcome-based Werewolf/avalon evaluations cannot see.
Controlled, leak-resistant protocol at scale
40 configurations from 29 checkpoints (1B–120B) receive identical pre-collected contexts; games finish before elicitation (B.1); outputs are JSON-schema constrained to the 7-point scale; family-specific reasoning parsers separate CoT; completion rates reported (≥95% target, 5 configs below 99%, A.3).
Right ablations for the obvious confounds
Honest scoping and a frontier reality check
Verified annotation
Weaknesses
Self-reported shifts with the prior fed back — anchoring untested
Beliefs are never linked to behavior
Statistical treatment ignores structure and multiplicity
Corpus homogeneity limits external validity
Frontier evidence is aggregate
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This benchmark paper targets a real gap: social-deduction evaluations of LLM agents (Werewolf, Avalon) typically reduce communication quality to win rate. The authors instead measure belief updating directly: for 1,224 annotated suspicion/accusation messages from 200 LLM-played 7-player Werewolf games, an observing village-side model states prior beliefs about the accuser and target on a −3..+3 scale, then states its shift after the message (§3). Central claims: (1) prior discrimination of true wolves improves with scale (ρ=0.54, p=4×10⁻⁴, §4.1); (2) accusations shift suspicion toward the target (avg +0.98 for village-side accusers, +0.74 for wolf accusers, Table 1) and away from the accuser; (3) trust dominates content — trusted accusers move beliefs nearly identically whether village-aligned (0.94) or wolf-aligned (0.96), Table 2; (4) larger open-weight models better exploit prior distrust (Distrust-bin wolf-accuser target shift 0.30→−0.12 from ≤4B to ≥30B, ρ=−0.37, §4.3). The abstract correctly scopes the credulity finding to open-weight models ≤120B.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper's Werewolf belief-shift benchmark measure?
It measures how accusation and suspicion messages change an observing LLM's stated beliefs during Werewolf games. For each of 1,224 annotated messages from 200 LLM-played games, the model first rates how wolf-like the accuser and the accused target seem on a −3 to +3 scale, then reports how those ratings shifted after seeing the message. This probes belief updating — a communication skill — rather than final win/loss outcomes.
Do LLMs blindly believe accusations in Werewolf?
Largely yes for the open-weight models tested (1B–120B): accusations shift suspicion toward the accused target and away from the accuser even when the accuser is actually a wolf (average target shift +0.74) or already distrusted (average +0.14). The effect is strongest when the accuser is trusted. Six frontier models partially break the pattern, discounting accusations from wolf-side accusers.
Which models were evaluated and is the data public?
29 open-weight checkpoints in 40 configurations: Qwen3/3.5/3.6, GPT-OSS-20B/120B, Gemma 4, Llama 3, Mistral/Ministral 3, Olmo 3/3.1, Phi-4, GLM-4.7-Flash, and Nemotron Nano, plus a preliminary set of six frontier models. The annotated game logs and evaluation code are released via the link in the abstract (rlg.iis.sinica.edu.tw/papers/werewolf-accusation-benchmark).
What are the main caveats about this study's conclusions?
Three stand out. Belief shifts are self-reported with the model's prior fed back into the prompt, so anchoring could inflate the measured shifts (acknowledged but not tested). All dialogue is LLM-generated within a single 7-player Werewolf environment, so transfer to human play or other settings is unknown. And the scale-trend statistics include a marginal uncorrected result (ρ=−0.37, p=0.018) carrying one of the headline claims.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 were extracted interleaved with body text; figure-derived values cited here (e.g., the 0.80→1.3 separation, ρ=0.54) were cross-checked against the prose that restates them, but visual figure details could not be independently verified.
- The header row of Table 6 extracted in garbled order; the frontier values cited were reconciled against the Appendix B.4 prose before use. No evidence of true text truncation — the extraction runs through the full references and Appendices A–B including Table 9.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.