(Whose defaults?) Is artificial intelligence reorienting archaeological methods?
Lorenzo Cardarelli, Roberto Ragno
cs.CY · cs.AI · cs.CL · cs.HC
Quick verdict
Tests if LLMs narrow archaeological methods: model advice is far narrower than the literature, but a 119k-abstract corpus shows no convergence yet; credible.
Summary
This preprint tests the 'mean-collapse' hypothesis: that generative AI, and vibe coding in particular, narrows the range of computational methods in archaeology. The authors analyse 119,327 Scopus abstracts (546 archaeology journals, 2010–2025), extract reported methods with a locally run Qwen3.5-9B model, and organise them into 25 broad categories and 241 data-driven clusters. A Bayesian hierarchical Dirichlet-multinomial regression finds a small but credible compositional shift after 2023, about 2.35 times smaller than the field's existing heterogeneity; no single technique changes credibly and the effective number of methods rises from 87.6 to 111.2. In a controlled experiment, Qwen3.5-9B and Gemma 4 E4B recommend methods for 28 standardised research questions under novice, intermediate and expert prompting; recommendation diversity is roughly one-third of the literature's (one-quarter for novices), and a negative binomial regression shows pre-2023 prevalence predicts recommendation frequency most strongly for novices (e.g. 1.039, 90% CI 0.810–1.275). The authors conclude a homogenising pressure is latent in the models but not yet visible in publications. Key limitations: only two small local models at a single low temperature, no human validation of the LLM extraction shown, and a diversity comparison that does not control for the very different sample sizes of the two streams.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- After ChatGPT's release, archaeological method composition shifted only slightly: the post-2023 shift is roughly 2.35 times smaller than the field's existing heterogeneity, no single technique changed credibly, and effective method diversity rose from 87.6 to 111.2.
- Local LLMs recommend method sets about one-third as diverse as the published literature, dropping to about one-quarter under novice prompting.
- A method's pre-2023 prevalence in the literature predicts how often LLMs recommend it, most strongly under novice prompting (Qwen3.5-9B: 1.039, 90% CI 0.810–1.275).
- The homogenising pressure is therefore latent in the models rather than visible in publications, and the design cannot establish causation.
Strengths
Documented, reproducible pipeline at scale
119,327 articles from 546 ASJC-1204 journals; extraction prompt reproduced verbatim (S4.1); six audited taxonomy phases; corpus flow fully tabulated (S3.3: 119,327 → 8,404 articles with methods → 17,229 pairs → 8,398); a verification script re-derives every ESM count from released artefacts.
Honest two-outcome design
Section 2 pre-states both possible outcomes; §5 reports the null bibliometric result against the authors' own hypothesis and offers three competing explanations (measurement stage, adopter composition, AI as diversifying force) instead of forcing convergence.
Bayesian rigour with conservative QC
Precise theoretical mechanism
Weaknesses
Unvalidated extraction on a 7% yield
Sample-size-confounded diversity comparison
Narrow model scope
Taxonomy noise with duplicate labels
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether generative AI — specifically 'vibe coding' — is narrowing the range of computational methods in archaeology, formalised as a mean-collapse hypothesis: LLMs trained on the aggregate literature should statistically favour canonical methods, progressively homogenising practice. Two interlocking streams test this: (i) a bibliometric analysis of 119,327 cleaned Scopus abstracts (546 ASJC-1204 journals, 2010–2025) with LLM-extracted method mentions, and (ii) a controlled prompting experiment in which two local models recommend methods for 28 standardised research problems under three knowledge profiles. The central claims: (1) method composition shifted only slightly after 2023 — the shift is ~2.35× smaller than the field's existing heterogeneity, no single technique changed credibly, and the effective number of methods (inverse Simpson) rose from 87.6 to 111.2; (2) LLM recommendations are only ~1/3 as diverse as the literature (1/4 under novice prompting); (3) pre-2023 prevalence predicts recommendation frequency, most strongly for novices (Qwen3.5-9B: 1.039, 90% CI 0.810–1.275); (4) convergence pressure is therefore latent, not yet visible — with an explicit admission that the design cannot establish causation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did the study find that ChatGPT or other LLMs have already narrowed archaeological methods?
No. Across 119,327 Scopus abstracts (2010–2025), methodological diversity increased after 2023 (effective number of methods 87.6 → 111.2) and no single technique changed credibly. The narrowing appears only in what the tested LLMs recommend, not yet in published practice.
What is the 'mean-collapse hypothesis'?
The idea that LLMs, trained on the aggregate of existing literature, statistically favour the most common methods when asked for advice, so researchers who rely on their defaults converge on a narrower set of techniques than the field has historically used.
Which AI models were used, and is the study reproducible?
The authors ran only local open-weight models: Qwen3.5-9B-Instruct and Gemma 4 E4B on a single 16 GB GPU. Code, all prompts, stimuli and full outputs are released at github.com/robertoragno/vibe-coding-archaeology; only the raw Scopus metadata cannot be redistributed for licensing reasons.
How many abstracts actually contained computational methods?
Only 8,404 of 119,327 cleaned articles yielded at least one extracted computational method, producing 17,229 article–method pairs (about 2.05 methods per article).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with explicit '[...sampling gap...]' markers: the detailed §4 results narrative, the reference list, and the bodies of S6–S12 (workflow checks, sensitivity analyses, concentration details) are not visible; conclusions about those areas rest on the abstract, §5 discussion and the listed ESM table of contents.
- The excerpt ends mid-way through Table S5.2 (at cluster L3-178); later L3 clusters and remaining supplementary content are not visible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.