Language-model groups overstate consensus when replaying human deliberation on a reasoning task
cs.AI · cs.CL · cs.CY · cs.MA
Quick verdict
Belief-anchored LLM groups overstate consensus vs the human Wason groups they replay (+34–44 pp); credible preregistered evidence with a post-hoc headline.
Summary
The paper asks whether large-language-model (LLM) agent groups reproduce the consensus structure of human deliberation. Using the public DeliData corpus of 500 Wason selection-task discussions, the author hash-split 400 calibration and 100 preregistered held-out groups, then replayed each held-out group with one belief-anchored agent per participant, seeded from that participant's pre-discussion answer, across three scaffolds, three seeds, and two inference modes of one pinned deepseek-v4-flash build, scoring humans and agents with identical code. Human full consensus ranged from 24.0% (carry-forward) to 57.0% (active-only) depending on scoring, and about one fifth of human participants never posted, versus 0.2% lurking agents. In participation-matched lurker-free groups (n=45), agent full-consensus exceeded humans by +34.1 points (chat) and +44.4 points (reasoner); a submit-based route (n=98) converged within 0.5 points. The gap survived disabled early stopping, an isomorphic reparameterization removing the memorizable answer, and a Qwen3-14B cross-family check. On the isomorphic task the reasoning mode agreed nearly unanimously, mostly on wrong answers (74.0%), showing that simulated consensus does not track accuracy. An exploratory subjective-regime study (N=20) under-converged instead. Limitations: the headline estimate was adopted post-unblinding; one served build plus a single cross-family model; residual measurement asymmetry; and belief anchors leak task competence.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Belief-anchored LLM agent groups reach full consensus far more often than the human DeliData groups they replay: +34.1 points (chat) and +44.4 points (reasoner) in participation-matched lurker-free groups, with a submit-based route converging within 0.5 points.
- Human full-consensus rates range from 24.0% to 57.0% across four scoring/participation definitions, so the operationalization of consensus is part of the validity argument.
- Simulated consensus does not track collective accuracy: under an isomorphic reparameterization the reasoning mode reaches near-unanimous agreement, 74.0% of it on incorrect card sets.
- The over-convergence is not explained by early stopping, surface memorization of the canonical answer, or scaffold choice, and it extends to one cross-family model (Qwen3-14B).
- In a small exploratory subjective task the same replay under-converges, suggesting the direction of the distortion may be task-regime dependent.
Strengths
Leakage-resistant preregistered design
A fixed hash splits 500 DeliData groups into 400 calibration / 100 held-out; both identifier lists were SHA256-committed in the OSF preregistration before held-out cells were generated, calibration could tune only the S1 improvement rate, and an integrity audit verified split disjointness — a strong architecture against curve-fitting (Methods, Calibration section).
Measurement operationalization made explicit and quantified
Table 2 shows human full consensus at 24.0% (carry-forward), 52.0% (submit-based), 57.0% (active-only), 51.1% (lurker-free, n=45) on the same 100 groups; the paper's 'identical code is not identical measurement' argument is a standalone contribution beyond the headline effect.
Correct-vs-wrong consensus decomposition
Honest deviation and null-result reporting
Broad protocol robustness
Weaknesses
Headline estimate is post-unblinding
Thin model coverage for a validity claim
Belief anchor leaks substantially
Single confirmatory task; subjective contrast is non-inferential
Under-seeded isomorphic cells and unquantified residual asymmetry
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a construct-validity question: when LLM agents are instantiated from real participants' pre-discussion answers (belief-anchored) and replay real group deliberations, do they preserve the consensus structure of the human groups? Using 100 hash-sealed, preregistered held-out groups from the DeliData Wason-selection-task corpus, the author reports that agent groups reach full consensus far above human rates. Three claims carry the paper: (1) simulated groups over-converge (+34.1 pp chat, +44.4 pp reasoner in participation-matched comparisons); (2) human consensus itself ranges 24.0%–57.0% across scoring and participation definitions, making the operationalization part of the validity argument (Table 2); (3) simulated consensus does not track accuracy — under an isomorphic reparameterization the reasoning mode agrees near-unanimously, 74.0% of groups on wrong answers. Scope is explicitly bounded to one objective task, one pinned served build, and one cross-family check.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
When LLM agents are instantiated from real participants' pre-discussion answers and replay 100 human Wason-task group discussions, the agent groups end in full consensus far more often than the humans did — by about 34–44 percentage points even when participation is matched — so simulated groups overstate how much humans agree.
Does high LLM-group consensus mean the groups found the right answer?
No. On an isomorphic rewording of the Wason task that removes the well-known canonical answer, the reasoning mode still agreed almost unanimously but mostly on incorrect card sets (74.0% of groups agreed on a wrong answer), showing that simulated consensus is not evidence of collective accuracy.
Couldn't the consensus gap just reflect silent human participants?
Only partly. About 20% of human participants never posted, and scoring definitions move human full consensus between 24% and 57%. But even in groups where every member spoke, agents still exceeded humans by 34–44 percentage points, and a comparison based on actually submitted answers gave nearly identical gaps.
Is the evidence limited to one AI model?
Largely. The confirmatory runs use two inference modes (chat and reasoning) of one pinned DeepSeek served build; a single exploratory check with locally served Qwen3-14B showed the same over-convergence, but coverage across model families remains thin, and the hosted model's exact build cannot be guaranteed over time.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (Figs. 1–4, Supplementary Fig. S1) are referenced but not analyzable from the extracted text; numeric claims were cross-checked only against in-text values and tables.
- Supplementary Notes and Tables are only partially embedded; some robustness detail (e.g., per-cell isomorphic CIs, Study 2 permutation nulls) could not be independently verified.
- Reproducibility assessment relies on the authors' descriptions of the Zenodo deposit; the archive itself was not inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.