Mapping the Emerging Social Science of Large Language Models
Yi Yang, Xiao Jia, Zeyun Dong, Chenzhang Wang, Zhanzhan Zhao
cs.CY · cs.AI · cs.CL
Quick verdict
A thorough dual-corpus mapping (198 + 47,719 papers) yielding a stable 3-domain taxonomy of LLM social science; strong methods, but data release unverified.
Summary
This study organizes the rapidly growing social-science literature on large language models into a three-domain taxonomy: LLM as Social Minds (socially interpretable model behavior), LLM Societies (collective dynamics among LLM agents), and LLM–Human Interactions (how people perceive, use, and are affected by LLMs). The authors combine a curated corpus of 198 papers read in full with a field-scale corpus of 47,719 formally published papers from five bibliographic databases. In Study 1, MPNet embeddings with K-means clustering recover the three-domain structure stably under resampling (adjusted Rand index 0.952), and within-cluster LDA resolves 13 subcategories. K-means assignments match author full-text classifications for 77.78% of papers and LLM classifications for 75.25%, with disagreements concentrated near cluster boundaries. In Study 2, structural topic modeling yields 15 topics, 13 of which map onto the taxonomy; LLM–Human Interactions dominates with 78.02% of mapped topic mass, though highly cited A*-conference papers skew toward the other two domains. Main limitations: weak absolute cluster-separation metrics, a diffuse third domain (60.5% correspondence), small venue-stratified subsets (n=89–137) without confidence intervals, and no code/data release visible in the text.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The social science of LLMs is organized into three recoverable domains — LLM as Social Minds, LLM Societies, and LLM–Human Interactions — resolvable into 13 LDA-derived subcategories.
- The three-domain structure is stable under resampling (mean ARI 0.952) and can be applied consistently by blinded authors (77.78% agreement with K-means, κ=0.662) and by an LLM evaluator using titles and abstracts only.
- The structure generalizes to field scale: 13 of 15 structural topics in a 47,719-paper corpus map onto the taxonomy, and K-means and STM domains agree for roughly three quarters of overlapping papers.
- LLM–Human Interactions dominates field volume (78.02% of mapped topic mass), but highly cited A*-conference papers concentrate in the other two domains (66.37%), revealing a journals-vs-conferences structural split.
- Disagreements among clustering, authors, and the LLM evaluator concentrate near semantic cluster boundaries, indicating domains that are distinguishable but permeable.
Strengths
Dual-corpus, multi-method validation battery
Three-domain structure is tested at two scales (198 curated + 47,719 field-scale papers) and by four routes: K-means resampling (mean ARI 0.952 across 100×90% draws, §2.1.1), 50 full-sample refits (1,225 pairwise ARIs, §5.1.2), blinded two-author full-text coding, and gpt-5.6-sol title/abstract classification (author–LLM agreement 85.86%, κ=0.781, §2.1.3).
Quantified boundary analysis instead of hand-waving
§2.1.3 localizes disagreement: the ten low-consensus papers show 20.0% vs 80.9% author–K-means correspondence (Fisher p<0.001), lower median silhouette (0.043 vs 0.135) and centroid margin (0.031 vs 0.148), i.e., disagreement sits at semantic boundaries — an unusually honest treatment of where the taxonomy fails.
Fully traceable field-scale funnel
Venue×citation stratification uncovers a real structural finding
Seed-level procedural documentation
Weaknesses
Weak and conflicting cluster-separation evidence for K=3
The volume-dominant domain is the least cohesive
Headline venue contrast rests on tiny strata without uncertainty
Reproducibility claim not yet cashed out
Model- and author-dependent corpus construction
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper organizes the social science of LLMs — defined as research treating LLMs or LLM-based agents as objects of social-scientific explanation, explicitly excluding LLM-as-instrument work (§1) — into a durable taxonomy. Its claims: (i) a three-domain structure (LLM as Social Minds; LLM Societies; LLM–Human Interactions) with 13 subcategories resolved by within-cluster LDA (Figure 1); (ii) recoverability by unsupervised clustering, blinded author full-text coding, and an LLM evaluator, with resampling stability (mean ARI 0.952); (iii) field-scale generality via structural topic modeling of 47,719 published papers, 13 of 15 topics mapping onto the taxonomy. The positioning is fair: prior syntheses cited in §1 are agenda pieces or domain-specific reviews, and a validated empirical map of the field is genuinely missing.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What are the three domains in the taxonomy of LLM social science?
The paper proposes: (1) LLM as Social Minds — socially interpretable behavior of individual models (reasoning/theory of mind, personality and bias, political and moral judgment, persuasion and deception); (2) LLM Societies — collective dynamics among interacting LLM agents (behavioral games, collective intelligence, group decision-making, large-scale simulation); (3) LLM–Human Interactions — how people perceive, use, and are affected by LLMs (trust, simulated empathy, work, creativity, education). Together they contain 13 subcategories.
How many papers did the study analyze and from where?
Two corpora: a curated set of 198 papers read in full by the authors (finalized September 2025), and a field-scale corpus of 47,719 formally published papers (2020–July 2026) distilled from 1,173,481 records retrieved from Semantic Scholar, OpenAlex, Scopus, PubMed, and Europe PMC using 41 prespecified LLM-related search terms.
How reliable is the three-domain classification?
Strong on stability, moderate on agreement. The K=3 clustering is highly stable under 90% resampling (adjusted Rand index 0.952) and matches blinded author full-text classifications for 77.78% of the 198 papers (κ=0.662). However, cluster-separation metrics are weak, agreement is uneven (only 60.5% for the large LLM–Human Interactions domain), and most disagreement clusters at semantic boundaries between domains.
What is the study's main finding about the field's composition?
By volume, LLM–Human Interactions dominates with 78.02% of domain-mapped topic mass versus 14.53% for LLM as Social Minds and 7.44% for LLM Societies. But among highly cited papers in top A* conferences the pattern reverses: Social Minds and Societies together hold 66.37%, while top-journal papers remain Human-Interactions-dominated at 76.81% — journals and conferences emphasize different parts of the field.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled 120k-character excerpt with marked [...sampling gap...] sections: parts of §2.2.3, §4 (discussion and limitations), the tail of §5.2, and most references were skipped; judgments touching those sections are partial and no authors' own stated limitations could be checked.
- No code or data availability statement appeared in the provided text; it may exist in skipped sections or the Supplementary Information, so codeAvailable=false reflects the excerpt only, not a verified absence.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.