From Process Loss to Assembly Bonus: Human-Grounded Diagnosis of Multi-Agent LLM Collaboration
Ala N. Tak, Teruhisa Misu, Kumar Akash, Zhaobo K. Zheng, Kevin H. Joo, Jonathan Gratch
cs.MA · cs.AI · cs.CL
Quick verdict
LLM groups mirror humans' assembly-bonus asymmetry yet diverge in process; rigorous matched study, but interventions rest partly on an oracle signal.
Summary
Comparing human group chats with protocol-matched multi-agent LLM deliberation on Wason-style deductive reasoning, this paper asks whether LLM groups succeed through human-like mechanisms rather than by outcome accuracy alone. Using DeliData's 500 human dialogues as baseline, the authors run three-agent LLM chats (Qwen3-4B/14B/32B, gpt-oss-20b; a GPT-5 subset) with intent-based floor-taking and voluntary exit, tracking solo-to-final trajectories over roughly 26,000 conversations, then test whether process signatures generalize to analogical (EAL), abductive (ACR), and analytical (AR-LSAT) tasks. Humans and LLMs show the same assembly-bonus asymmetry: discussion improves the average member more often than it preserves the best initial member. Mediation analysis (N=5,700) shows model-composition diversity acts entirely through initial-answer diversity. At process level, LLM groups follow majorities more (peer-pull +0.59 vs +0.35), surface less unique information, and lock in earlier; correct minority signals survive mainly when re-expressed early. Interventions — revealing solo submissions, majority-flip announcements, oracle confidence projection — yield only modest gains, and realistic confidence proxies prove miscalibrated. Limitations include no human baseline on the three generalization tasks, small human cells (n=8–56) in some analyses, and mostly three-agent groups with mid-size open models.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Humans and LLM groups share the assembly-bonus asymmetry: discussion improves the average member (pooled Wason WAB 50.8%) far more often than it exceeds the best initial member (SAB 20.1%), while process loss remains substantial (34.4%).
- Model-composition diversity affects collective outcomes only through initial-answer diversity, with full mediation (direct paths non-significant, Fig. 2).
- The main human-LLM differences are process-level: LLM groups show stronger peer conformity (Fig. 3), lower unique-information realization, and earlier consensus lock-in than humans.
- Correct minority signals survive mainly when publicly re-expressed early and carried to the final submission, and inference-time interventions (RevSol, MajFlip, oracle ConfProj) improve outcomes only modestly without removing the coordination bottleneck.
Strengths
Genuinely human-grounded baseline with affordance-matched protocol
DeliData (Karadzhov et al., 2023; 500 dialogues, mean 3.16 participants) anchors all Wason comparisons; the LLM setup preserves solo, intermediate, and final answers with intent-based floor-taking (priority bids 1-5) and voluntary exit (Sec. 2.1), enabling trajectory metrics absent from endpoint-accuracy debate studies.
Metric suite grounded in group-psychology theory, not just accuracy
Collective gain, BMG, WAB, SAB, PL, Recovery, and Drift (Sec. 2.2) operationalize Steiner (1972) and social decision-scheme theory; Table 1 exposes patterns a final-accuracy view would hide, e.g., Qwen-14B homo pairs high recovery (54.5%) with high drift (62.0%).
Clean mediation result with design implications
Unusual robustness breadth and honesty about failure of realistic proxies
Process mechanisms quantified, not just asserted
Weaknesses
Human baseline exists only for Wason
Headline rates reported without uncertainty
Very small human cells in the diversity analysis
Comparability of scoring and prompts between populations is assumed
Heterogeneous rows pool compositions opaquely
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether multi-agent LLM deliberation succeeds or fails through *human-like* mechanisms, using human group chats as the diagnostic baseline rather than final accuracy alone. Three central claims: (1) humans and LLM groups share the assembly-bonus asymmetry — discussion improves the average member (WAB) more often than it preserves the best initial member (SAB); (2) model-composition diversity affects outcomes only through initial-answer diversity; (3) the populations diverge at process level — LLM groups follow majorities more (peer-pull +0.59 vs +0.35 for humans, Fig. 3), surface less unique information, and lock in earlier — and interventions help only modestly. Claims are appropriately scoped: the authors explicitly caution against equating outcome similarity with mechanism similarity (Secs. 1, 8).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What are 'assembly bonus' and 'process loss' in this paper?
Assembly bonus is when group discussion yields a better answer than members started with; the paper distinguishes weak assembly bonus (the average member improves) from strong assembly bonus (the group exceeds its best initial member). Process loss is the opposite failure: discussion ends below what the best initial member already knew. On Wason, weak assembly bonus occurred in 63.8% of human groups and roughly 30-60% of LLM runs, while strong assembly bonus was 18.4% for humans and varied widely (6.6-54.5%) across LLM configurations.
Which models were used and how large is the study?
Three-agent groups built from Qwen3-4B, Qwen3-14B, Qwen3-32B, and gpt-oss-20b in homogeneous and heterogeneous mixes, plus a subset of runs containing GPT-5. Across four reasoning tasks and intervention conditions the study covers roughly 26,000 LLM group conversations, compared against 500 human group dialogues from the public DeliData dataset.
Do multi-agent LLM discussions actually improve answer quality?
Partly. Discussion improved the average member's answer in about half of LLM runs (pooled Wason weak assembly bonus 50.8%) and collective gain was positive on all four tasks, but exceeding the best initial member was rare (pooled SAB 20.1%), and in about a third of Wason runs the group lost its best starting answer (process loss 34.4%). Discussion is therefore not reliably better than simply keeping the best solo answer.
Is code and data available for this paper?
The paper links a code repository (github.com/ala-ntak/AgentDelibU). The human baseline is the public DeliData dataset, and most models (Qwen3 family, gpt-oss-20b) are open-weight; however, part of the analysis uses GPT-5 via API, and the authors state that full reproduction requires about two weeks on A100/H100/H200-class GPUs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Inequality symbols were lost in figure-legend extraction (e.g., Fig. 5a legend reads 'Human BMG 0 (n=314)', evidently 'BMG <= 0'); interpretations involving such symbols were inferred from context, not the glyph itself.
- The provided text ends naturally after Appx. M (compute resources) with no mid-sentence cutoff, so no truncation is assumed; only figure-glyph loss is noted.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.