Citing Less Critically: LLMs Reshape the Rhetoric and Reach of Scientific Citation
Yixuan Liu, Lin Chen, Zhuoqi Liu, Jianglin Lu, Dakota Murray
cs.DL · cs.CL · cs.CY · cs.SI
Quick verdict
Paper claims LLMs cite less critically and more socially distant than humans; masked-citation audit of 6 LLMs is rigorous with judge/coverage caveats.
Summary
The paper asks whether LLMs reproduce human citation rhetoric. Using a masked-citation task, the authors mask each citation sentence in 1,746 ACL/EMNLP/NAACL 2025 papers (63,944 contexts, 132,913 slots) and prompt six LLMs (GPT-5.1, Claude-3.5-Haiku, Gemini-2.0-Flash, DeepSeek-V3.2, Llama-4-Maverick, Qwen2.5-72B) to reconstruct it, producing position-aligned counterfactual citations. An LLM-as-judge (Gemini-3-Flash, validated against three human annotators, κ=0.60) labels intent, and a 2.1M-node, 20.3M-edge coauthorship network measures citing-cited distance. Three results: (1) LLMs under-produce contrasting citations (10.7–17.2% vs 19.3% human) and preserve only 34.6–50.6% of human critical contexts; (2) popularity/recency/team-size biases are intent-modulated — the citation-count gap peaks at supporting (up to 4.45×), recency at contrasting (up to +3.27 years), team size at mentioning (down to 0.30×); (3) humans cite in-network (d≤1 at 7–10%, elevated for supporting) while LLMs stay below 1.6% with flat intent gradients. Robustness checks include a second judge, shared-context intersection, wider context windows, live web search, and a pre-2024 restriction. Limitations: both judges are LLMs (human validation only 90 sentences), Dimensions match rates vary 39.5–81.9% across models, the network reaches only 26.5% of human slots, and scope is limited to English NLP papers.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLMs systematically under-produce contrasting citations (10.7-17.2% vs human 19.3%) and rewrite about half of human critical citation contexts into warmer intents.
- LLM citation biases are intent-modulated: the cited-impact gap peaks at supporting citations (up to 4.45x), the recency gap at contrasting (up to +3.27 years), and the team-size gap at mentioning (down to 0.30x).
- Humans disproportionately cite socially proximate authors (in-network d≤1 at 7-10%, elevated for supporting intent), while all six LLMs stay below 1.6% with essentially no self-citation and flat intent gradients.
- The warming persists under a second judge, wider context windows, live web search, and human annotator labels, indicating a generation-stage phenomenon rather than a retrieval failure.
Strengths
Slot-level counterfactual design isolates citation choice
Section 2.1: every LLM citation is a counterfactual to the human choice at the same slot, with a post-cutoff corpus preventing retrieval of the held-out citation and count-matching removing length confounds — a genuinely stronger design than the flat retrieval audits it critiques (Algaba 2025/2026; Tian 2024).
Exceptionally broad robustness battery
Second independent judge (Appendix C, inter-judge κ=0.52/0.45, Table 5), shared-context intersection (Appendix H: 12,556 contexts, intent-amplified pattern reproduced, e.g., supporting citation-gap 3.68-4.74x), context-window widening (Appendix F: contrasting 10.0-10.9%), live web search (Appendix I: contrasting stays ~10%), pre-2024 restriction (Appendix G), and within-field distance control (Appendix J: +0.31 to +0.39 hops, p<10^-6).
Human validation shows warming is not a judge artifact
Transparent error and hallucination accounting
Honest, specific limitations
Weaknesses
Residual judge circularity with thin human ground truth
Differential attrition across models
Network coverage asymmetry threatens the distance comparison
Recency finding is entangled with the training cutoff
No repeat-generation variance reported
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether LLM-assisted writing changes not just *what* gets cited but *how* — the rhetorical stance (supporting / contrasting / mentioning) and the social reach of citation choices. Its masked-citation audit covers 1,746 ACL/EMNLP/NAACL 2025 main-track papers with arXiv .tex/.bib (63,944 contexts, 132,913 slots); six LLMs (GPT-5.1, Claude-3.5-Haiku, Gemini-2.0-Flash, DeepSeek-V3.2, Llama-4-Maverick, Qwen2.5-72B) reconstruct each masked citation sentence in place. Three claims: (1) LLMs "warm" citation tone, under-producing contrasting citations (10.7–17.2% vs human 19.3%) and preserving only 34.6–50.6% of human critical framings under the primary judge (20.3–32.4% under the second judge); (2) biases are intent-conditioned — cited-impact gap peaks at supporting (to 4.45×, Fig. 3a), recency at contrasting (to +3.27 yr, Fig. 3b), team size at mentioning (to 0.30×, Fig. 3c); (3) human citations are socially proximate (in-network d≤1 at 7–10%, supporting 9.8% vs contrasting 7.3%) while every LLM sits below 1.6% with near-zero self-citation. Claims are properly scoped to English NLP papers in the Limitations section.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'LLMs cite less critically' mean concretely?
In the same masked citation slots from 1,746 ACL/EMNLP/NAACL 2025 papers, human sentences were judged contrasting 19.3% of the time, while LLM-filled versions were judged contrasting only 10.7-17.2% across six models; roughly half of human critical framings were rewritten as mentioning or supporting, and this warming reappears in human annotators' own labels.
Which LLMs were tested in this study?
GPT-5.1, Claude-3.5-Haiku, Gemini-2.0-Flash, DeepSeek-V3.2, Llama-4-Maverick, and Qwen2.5-72B-Instruct, each prompted at temperature 0 to reconstruct masked citation sentences in place, with the human choice held out as a counterfactual.
Do LLMs cite the writing authors' own papers or close collaborators?
Almost never. Human citations to self or direct coauthors (distance d≤1) are 7-10% of slots, highest for supporting intent; every LLM stays below 1.6% with self-citations essentially absent, and their mean coauthorship distance is shifted right by 0.25-0.49 hops.
Is the result just because models don't know the cited papers (training cutoff)?
Not entirely: the warming persists when GPT-5.1 gets live web search (contrasting stays ~10%, Appendix I), and the contrasting-intent recency gap survives restricting to pre-2024 works (Appendix G). The authors still flag the post-cutoff corpus as a limitation for absolute recency claims.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Reviewed from a sampled excerpt: middle sections marked [...sampling gap...] (parts of Methods, the Section 5 opening, Discussion, references, and Appendix A examples are not visible); all numbers cited here come from visible text.
- Figures 2-4 are available only as extracted caption/axis text; visual details (CI widths, density shapes) could not be inspected.
- Whether temperature-0 generations were repeated per slot is not discussed in the visible text; such a discussion may exist in unseen sections.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.