Social Influence and the Allocation of Scientific Attention in AI Populations
Maxim Chupilkin, Maxim Chupilkin
cs.AI · econ.GN
Quick verdict
Music Lab in LLM agents: social counts cut selections 17.2%, random head start adds 45.55pp; elegant design, limited by single model and tiny community sample.
Summary
This paper adapts the Salganik–Dodds–Watts Music Lab design to academic attention among large language model agents. In Experiment 1, 1,000 GPT-5.6 Sol invocations, arranged in ten communities of 100 sequential agents, chose which of all 114 regular 2025 American Economic Review articles to 'read in full' from titles and abstracts, with unrestricted choice; five communities saw only the menu, five saw cumulative within-community selection counts. Social information cut mean selections from 17.29 to 14.32 papers per agent (−17.2%), arm-wide coverage from 90 to 73 papers, and raised mean within-community Gini from 0.753 to 0.844, with greater between-community variation. Paper rankings correlate at 0.953 across conditions, so social information concentrated attention on already-favoured works, yet early community-specific advantages persisted (a 10-percentage-point early lead predicts a 3.09-point later advantage socially versus 0.19 independently), and thinning to equal selection volume leaves the gaps intact. In Experiment 2, five initial selections were randomly assigned to 20 eligible papers across 20 communities of ten agents in a balanced paired design: boosted papers were subsequently selected at 60.00% versus 14.45% (45.55 points, 95% CI 41.20–49.90, randomisation p<0.001), positive in all ten pairs. Choices correlate modestly with OpenAlex citations (0.243–0.288) and barely with RePEc downloads (0.027–0.044). Main limitations: a single model version left unpinned, only five communities per arm, cost-free selections with no reading budget, and data/code promised only at publication.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Visible social information reduces AI agents' selection volume by 17.2% (17.29 to 14.32 papers per agent) and narrows collective coverage from 90 to 73 papers across five communities.
- Social information concentrates attention (mean within-community Gini 0.753 to 0.844) and increases between-community variation, and both effects persist when selection volume is equalised by thinning.
- A randomly assigned five-vote head start causally raises subsequent selection of the same papers by 45.55 percentage points (60.00% vs 14.45%, all ten paired contrasts positive).
- Agent choices correspond only modestly with OpenAlex citations (Spearman 0.243–0.288) and weakly with RePEc downloads (0.027–0.044), indicating an attention allocation distinct from these human-activity benchmarks.
Strengths
Strong causal identification in Experiment 2
The balanced cross-over design boosts each of 20 eligible papers in ten communities and not in ten, with fresh random splits per pair, seeded selections excluded, and the analysis plan fixed before collection (§4.1); within-paper contrasts neutralise abstract-level appeal, a 100,000-draw randomisation test yields 45.55pp [41.20, 49.90] with p<0.001, and all ten pair contrasts are positive (Appendix D).
Confound-aware Experiment 1
The first-visit comparison (all social counts zero) shows no volume difference (−2.00 papers, 95% CI −6.97 to 2.97, p=0.326, §3.1), and hypergeometric thinning to a common 1,404 selections (2,000 draws) preserves the Gini gap (0.754 vs 0.844) and total-variation gap (0.087 vs 0.107) (Appendix C.2), ruling out selection volume as the driver of concentration.
Inference at the correct level, with honest scope statements
Unusually complete audit trail
Honest external-benchmark reporting
Weaknesses
Single model with an unpinned version
Very few independent units in Experiment 1
The volume effect's mechanism is not adjudicated
No dose-response for the popularity boost
External benchmarks lack a human comparator
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Chupilkin (Oxford) adapts the Salganik–Dodds–Watts Music Lab design (2006) to 'a market for academic attention' populated by LLM agents. The claims: (1) showing agents cumulative selection counts cuts per-agent selections by 17.2% (17.29 to 14.32 of 114 AER 2025 papers, Table 1) and shrinks collective coverage (90 to 73 papers); (2) social information raises concentration (within-community Gini 0.753 to 0.844) and between-community variation; (3) a randomly assigned five-vote head start causally lifts subsequent selection of the same papers by 45.55 points (60.00% vs 14.45%); (4) agent attention maps only weakly onto citations (Spearman 0.243–0.288) and barely onto downloads (0.027–0.044). The framing — information rules as institutions governing AI attention — is timely, and the claims are mostly calibrated to what the experiments actually show.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What did the AI agents see when choosing papers?
Each agent received only the titles and complete abstracts of all 114 regular research articles from the American Economic Review's 2025 volume, in a random order; author names, publication dates, the journal label, citation counts and download statistics were withheld. Agents in the social-influence condition additionally saw how many earlier agents in their own community had selected each paper.
How big was the artificial popularity effect?
In the randomised experiment, papers given five initial selections were subsequently chosen at 60.00% of opportunities versus 14.45% for the identical papers starting at zero — a 45.55 percentage-point difference (95% CI 41.20–49.90, randomisation p<0.001), with the advantage positive in all ten community pairs and persisting among the last five agents.
Did the AI agents just pick the most-cited papers?
No. Aggregate choices correlate only modestly with OpenAlex citation counts (Spearman 0.243–0.288 depending on condition) and hardly at all with RePEc download counts (0.027–0.044), suggesting the agents' collective attention differs substantially from these human-mediated indicators.
Can the experiments be reproduced?
The paper states that data and code will be made publicly available with publication, and that saved prompts, responses and validation records already allow full reconstruction of every choice history. Note the model is identified only as 'gpt-5.6-sol' without a dated version, so exact replication depends on API availability and version stability.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (the Fig. 4 heatmap, Fig. 3/C1 trajectory plots) appear in the extraction only as garbled axis and label text; visual patterns could not be independently verified.
- Full API settings, presentation schedules and snapshot dates are said to reside in replication records not included in the text.
- Several 2026-dated references (GPT-5.6 Sol, CiteAgent, Robin, the AEA–Refine partnership) could not be verified offline.
- The extraction appears complete from abstract through Appendix D; no missing-section truncation was detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.