Diverse Minds, Divided Networks? Personality Composition, Polarization, and Collective Intelligence in LLM-Based Social Simulations
physics.soc-ph · cs.CL
Quick verdict
Across 991 LLM-agent sims, trait diversity spreads opinions but cuts segregation; no accuracy trade-off; rigorous; interaction rests on replication models.
Summary
The paper introduces TraitMix, a simulation framework treating the Big Five personality composition of 100-agent LLM societies — both mean trait levels and trait heterogeneity — as a controlled experimental variable, measuring polarization and collective intelligence in the same runs. Across 991 simulations on one RTX 4090 (Llama-3.1-8B primary; five replication models; six contested topics plus a neutral filler control), three results emerge. First, heterogeneity is the strongest compositional lever: realised trait dispersion correlates with opinion variance at r=+0.97, and diversity raises dispersion while lowering segregation — homogeneous societies are consensual echo chambers, and the human-calibrated composition is the most insular condition and the only one with significant collective-accuracy deficits. Second, Agreeableness moderates the sign of Openness's effect on opinion variance; the interaction replicates in Qwen2.5-14B and 32B, though the primary model's fit is influence-driven (two cells exceed Cook's distance 1; significance survives 3/9 leave-one-out fits). Third, no polarization measure predicts poorer collective performance; cross-cutting interaction associates positively with accuracy (r=+0.41) and survives partialling on the diversity-prediction identity. Trait-level main effects replicate at near-chance rates across models, Qwen2.5-7B shows neither joint-trait finding, and the alignment evidence is between-composition only; collective-intelligence conclusions rest on a single working measure because the hidden-profile task floors near 0.12.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Trait heterogeneity is the strongest compositional variable: it raises opinion dispersion while lowering segregation, so homogeneous LLM societies become consensual echo chambers rather than moderate ones (realised trait SD vs opinion variance r=+0.966).
- Agreeableness determines the sign of Openness's effect on opinion variance — a crossover interaction replicating in Qwen2.5-14B and 32B, though the primary model's estimate is influence-driven.
- No polarization measure predicts poorer collective performance; cross-cutting interaction is positively associated with collective accuracy (r=+0.409) and is the only measure whose association survives partialling on the diversity-prediction identity and realised trait dispersion.
- A human-calibrated composition built from published IPIP-NEO-120 norms is the most insular condition in the study and the only one with significant collective-accuracy deficits.
- Trait-level main effects largely fail to replicate across models (sign agreement 4/10 to 6/10); what transfers across families is the moderation structure, not magnitudes.
Strengths
First dual-outcome design: composition manipulated, both outcome families measured in the same runs
Table 1 shows no prior agent-society study both manipulates Big Five composition and measures polarization + collective intelligence together; E1 (trait levels), E2 (heterogeneity) and E3 (O×A response surface) isolate level from spread, which prior 'assigned personality' work could not test.
Pre-specified, unusually honest statistical protocol
Section 4.5: paired t-tests with Holm–Bonferroni within family×metric, dz/Hedges' g/Cliff's δ with 10,000-resample bootstrap CIs on every contrast, an explicit statement of the Wilcoxon p-floor (0.0078 at n=8), a seed-extension rule, and within-model standardisation to keep released numbers recomputable.
Withdraws its own findings when checks fail
Circularity ablations and robustness audit beyond standard practice
Complete artefact release at consumer-GPU scale
Weaknesses
Headline interaction is fragile in the primary model
Primary-model trait-level effects shown despite near-chance cross-model replication
Collective-intelligence conclusions rest on one working measure
Alignment association is between-composition and mechanistically unresolved
Structured cross-model anomaly the design cannot explain
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
TraitMix targets the junction of two literatures that rarely meet: LLM opinion-dynamics studies (echo chambers, no group tasks) and LLM collective-intelligence studies (task teams, no networks or opinions). Table 1 formalises this scope claim and it is fair: no listed prior study both manipulates Big Five composition and measures both outcome families in one system. Composition — trait means μ and heterogeneity Σ — is the independent variable across 991 runs of 100-agent societies (Llama-3.1-8B primary, five further models), six contested topics plus a neutral filler. The paper's defining feature is disciplined self-correction: it withdraws its own raised-trait asymmetry finding (Section 5.2) after the six-topic replication, and withdraws three of four polarization→accuracy associations after partialling on the aggregation identity (Section 5.8).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TraitMix?
TraitMix is an experimental design for LLM-based social simulation in which the Big Five personality composition of a 100-agent simulated society — both average trait levels and trait diversity — is systematically varied, while polarization and collective task performance are measured in the same simulation runs. The study reports 991 simulations across six language models, with code, configurations and run-level data released publicly.
Does reducing polarization hurt collective intelligence in LLM agent societies?
In this study, no trade-off was found: no polarization measure predicted poorer collective performance. Cross-cutting interaction — replies between agents holding opposing opinions — was positively associated with collective estimation accuracy (r≈+0.41), and the association survived statistical controls for the mechanical link between estimate diversity and average error. The evidence is correlational and operates between compositions, not within them.
Does personality diversity increase or decrease polarization among LLM agents?
It depends which face of polarization is measured. Personality-diverse societies held more dispersed opinions (more polarization-as-dispersion) but were less segregated into camps (less polarization-as-segregation). Homogeneous societies were not moderate; they formed consensual echo chambers with low opinion variance and high within-camp closure.
Did the findings replicate across different language models?
Partially. The Openness-by-Agreeableness interaction replicated in Qwen2.5-14B and Qwen2.5-32B, and the heterogeneity effects held in Llama-3.1-8B and Qwen2.5-14B. But trait-level main effects agreed across models at near-chance rates, Qwen2.5-7B showed neither joint-trait finding, and two smaller models were excluded by a pre-specified personality-induction validation gate.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with three marked [...sampling gap...] regions: parts of Section 5.1/5.2, the middle of Section 5.8 (exact partialling specification) through Section 5.10, part of Section 6.5, and portions of the supplementary ledger were skipped; judgments about those passages rely on the paper's own cross-references visible in the retained text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.