The Wisdom of Artificial Deliberative Crowds
Federico Barrera-Lemarchand, Mariano Sigman, Joaquin Navajas
cs.AI
Quick verdict
Across 4 domains deliberating LLM crowds beat averaging only with model diversity; paired evidence strong but bootstrap p-values inflated, code unreleased.
Summary
The paper tests whether the human 'wisdom of deliberative crowds' effect—where averaging consensus estimates from small deliberating groups beats pooling many independent judgments—transfers to large language models. Three small models from different families (Claude Haiku 4.5, GPT-5-nano, Gemini 3.1 Flash Lite) followed an identical three-stage protocol (independent answers, ten-round chat deliberation in triads, private re-answer) across four domains: jar-count estimation, NeurIPS 2024 peer review against editorial verdicts, detection of sabotaging agents in the MRT safety corpus, and 2026 FIFA World Cup forecasting benchmarked against Polymarket. In all four studies, group consensuses were more accurate than pooled independent answers, and post-deliberation individual answers retained the gain. Crucially, groups of three clones of one model showed no deliberation benefit (Bayes factors 3.99-6.19 favoring the null), isolating model diversity as the active ingredient, and a single dominant model beat the deliberative crowd in three of four domains. The deliberative crowd matched Polymarket (BF01=3.63) despite the market's information advantage. Limitations: one small model per family from a single generation; p-values computed over bootstrap resamples are inflated by pseudo-replication; Study 3 rests on 25 sabotage trajectories with floor effects on benign cases; post-deliberation individuals largely copied the consensus; the noisy editorial criterion and possible training-data overlap in Study 2; code and data promised on OSF but not yet released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- In all four domains, averaging consensus estimates from diverse LLM triads is more accurate than classical wisdom-of-crowds aggregation of independent answers.
- Post-deliberation individual answers retain the collective gain, exceeding the consensus in Study 1 and matching it in Studies 2-3.
- Model diversity is the active ingredient: groups of three clones of one model show no deliberation benefit, with Bayesian evidence supporting the null.
- Expert dominance appears in three of four domains — a single best model beats the deliberative crowd, and which model dominates is domain-specific and unknowable ex ante.
- A deliberating crowd of three small models matches a real-money prediction market on 2026 World Cup forecasting despite a large information disadvantage.
Strengths
One protocol, four domains
The same three-stage chat deliberation (≤10 rounds, ~7-word messages, tagged consensus) is held constant across physical estimation (Study 1), expert evaluation (Study 2), AI-safety monitoring (Study 3) and pre-event forecasting (Study 4), so the replications jointly test generality rather than task-specific tuning; convergence was near-universal (167/168, 600/600, 299/300, 175/176 groups).
Clone manipulation cleanly isolates diversity
Tripling a single model (n=56 groups per family in Study 1) removes diversity while holding protocol, task and agent count fixed; Bayesian analyses support absence of effect (BF01=5.60 Anthropic, 3.99 OpenAI, 6.19 Gemini) against a significant size-matched heterogeneous control (z=3.61, p=3.1e-4, d=0.48) — a manipulation impossible in human crowds.
External, incentive-backed benchmark
Item-level statistical hygiene
Honest, specific limitations
Weaknesses
Bootstrap p-values are pseudo-replicated
Stage i2 'individual improvement' is mostly consensus adoption
Three models do not make a crowd of families
Missing matched-compute baseline
Study 3 evidence is thin where stakes are highest
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether the human 'wisdom of deliberative crowds' (Navajas et al., 2018; Barrera-Lemarchand et al., 2026) — averaging consensus estimates from small deliberating groups beats pooling many independent judgments, and individuals improve after deliberating — transfers to LLMs. Three small models, one per provider (claude-haiku-4-5-20251001, gpt-5-nano, gemini-3.1-flash-lite), run an identical three-stage protocol (i1: independent answers with short justifications; c: triads debate in a simulated chatroom, up to 10 rounds of ~7-word messages, tagged consensus; i2: private re-answer) across four domains: jar-disk estimation (14 photos, truth 38-589), NeurIPS 2024 peer review (100 token-matched submissions, 50 oral-accepted vs 50 rejected, anonymized), sabotage detection in 50 MRT AgentCompany trajectories (25 selective_liar, 25 benign), and 2026 FIFA World Cup forecasting (44 events) against Polymarket. Central claims: (1) consensus beats classical aggregation in all four studies; (2) post-deliberation individuals retain the gain; (3) clone groups show no deliberation benefit — diversity is the active ingredient; (4) expert dominance: the best single model beats the crowd in three of four domains, with a domain-specific identity; (5) the deliberative crowd matches Polymarket.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'wisdom of deliberative crowds'?
It is the finding, first shown in human experiments, that if people estimate in small groups that try to reach consensus and you then average the group answers, the result beats averaging hundreds of independent individual guesses — and the individuals themselves become more accurate after discussing. This paper shows the same two signatures emerge when three different small LLMs deliberate in a simulated chatroom, provided the models come from different families.
Did the LLM crowd beat Polymarket on the 2026 World Cup?
No — it matched it. The deliberating crowd of three small models (Claude Haiku 4.5, GPT-5-nano, Gemini 3.1 Flash Lite) was statistically indistinguishable from Polymarket prices across 44 events (Bayes factor 3.63 favoring no difference), while the non-deliberating independent crowd was only marginally worse than the market (p=0.067). All forecasts were made before each round was played, and the models had no search tools, so the market's information advantage makes the equivalence a conservative result.
Why didn't groups made of the same LLM benefit from deliberating?
Because diversity drives the gain. When a group consisted of three copies of one model, its consensus was no better than averaging that model's own independent answers (Bayes factors 3.99-6.19 supporting no effect), whereas mixed groups with one model per family consistently improved. A single dominant model could even beat the mixed crowd, and which model dominates changes from task to task, so it cannot be identified in advance.
Which models were used in the study?
One small model per major provider: Anthropic's Claude Haiku 4.5, OpenAI's GPT-5-nano, and Google's Gemini 3.1 Flash Lite, called through their public APIs without fixed temperature or seeds. They were tested on jar disk-count estimation, NeurIPS 2024 peer review, detection of malicious AI agents (MRT dataset), and 2026 FIFA World Cup forecasting.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures appear only as captions in the extraction; curve shapes, bands and graphical magnitudes (e.g. Fig. 2A-5A) could not be inspected directly.
- The human counterpart study (Barrera-Lemarchand et al., 2026) and several 2026-dated references could not be verified from this text.
- OSF data/code are described as 'shortly available'; current availability is unconfirmed, so reproducibility claims rest on the paper's description.
- The extraction spans abstract through Appendix 1 with internally consistent sample sizes (504=3x12x14; 149=25x6-1; 175=44x4-1), supporting high confidence.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.