More than half of recent astronomy papers are written with language-model assistance
astro-ph.IM · cs.CL · cs.DL
Quick verdict
Claims 54% of 2025 astro-ph papers carry an LLM lexical trace via a calibrated Bayesian mixture; rigorous, but rests on an unobservable background extrapolation.
Summary
The authors estimate the share of astro-ph papers whose prose bears the lexical signature of language-model assistance, analyzing 207,111 full texts (2015–mid-2026) from an updated AstroMLab 5 catalog. They count 38 imported 'marker' words per paper, normalize by text length, and fit a hierarchical Bayesian mixture of negative binomials in which pre-2020 papers anchor the unassisted background and 392 papers disclosing model use anchor the assisted excess; the background past 2020 is extended under three explicit assumptions (linear, frozen, control-tracked). The primary (linear) specification gives 54 (+8/−8 stat, 95%) (+26/−0 sys)% for 2025, rising to 87 (+8/−10)% in 2026H1, and never below 36% across variants; only 0.81% of 2025 papers disclose. The fitted excess of assisted prose fades from 3.5× background (2023) to 1.5× (2026), which the declared-paper anchor separates from falling use. A corpus-derived 34-word basket yields 80% for 2025. The authors show document-frequency estimators fail null tests on full texts — neutral controls imply ~51% apparent prevalence — because papers lengthened. Key limitations: the declared-to-silent marker-rate ratio is assumed (doubling it moves the estimate +22 pp), background extrapolation is unobservable in principle, and latest-revision OCR may leak post-2022 text into the pre-2020 calibration. Code, feature tables and posteriors are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- 54 (+8/−8 stat, 95%; +26/−0 background systematic)% of 2025 astro-ph papers carry a lexical language-model trace under the primary linear-background specification, never below 36% across all tested variants.
- Only 0.81% of 2025 papers disclose model use, implying roughly 66 papers with a trace per declaration.
- The marker excess of assisted prose fades from 3.5× background in 2023 to 1.5× in 2026, and the model separates this fading from reduced use via the declared-paper anchor.
- Document-frequency (presence/absence) estimators fail on full texts because neutral control words alone imply ~51% apparent prevalence, driven by median paper length growing from ~4,500 to 6,400 tokens.
- By the first half of 2026 the traced fraction reaches 87 (+8/−10)%, approaching the ~89% reported for biomedical full texts in late 2025.
Strengths
Fixes a demonstrated failure of the standard estimator
Presence-based document-frequency arithmetic returns ~51% apparent prevalence on neutral control words (§4.2) because median paper length grew ~4,500→6,400 tokens; the count-given-length offset removes most of that control excess (Fig. 2).
Honest treatment of the unidentifiable background
Three explicit continuations (linear/frozen/control-tracked) reported separately with a one-sided +26/−0 pp systematic (§2.5, §3.1) rather than folded into one number; linear is chosen as primary precisely because it is most conservative.
Multi-layered validation
Separates fading trace from falling use
Transparency and reproducibility
Weaknesses
Calibration transferability is unidentifiable
Headline is conditional on an unobservable background
Revision leakage flagged but never quantified
Corpus-derived basket may absorb non-LLM drift
Late-2026 estimate sits where anchoring strains
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper estimates the fraction of astro-ph papers whose analyzed prose carries the lexical signature of language-model assistance. Corpus: 207,111 papers first submitted January 2015–June 2026, drawn from an update of the AstroMLab 5 catalog (433,176 papers through August 2026), restricted to ≥300 analyzed tokens after section-segmented cleaning (introduction/methods/results/discussion/conclusions; abstract, references, appendices excluded; median ~2,500 analyzed tokens). Headline claims: (1) 54 (+8/−8 stat, 95%) (+26/−0 sys)% of 2025 papers show a trace under the primary linear-background specification, ≥36% across all variants, and 87 (+8/−10)% in 2026H1; (2) only 0.81% of 2025 papers disclose use, ~66 traced papers per declaration; (3) the marker excess of assisted prose fades from 3.5× background (2023) to 1.5× (2026). The estimand is explicitly a floor: a lexical trace consistent with the declared (mostly grammar/clarity editing) component, not machine authorship.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What percentage of astronomy papers are written with ChatGPT or other AI tools?
This paper estimates that 54% of 2025 astro-ph papers carry a lexical trace of language-model assistance (95% statistical interval ±8 points, plus up to +26 points of background-systematic uncertainty), rising to 87% in the first half of 2026. A 'trace' means the prose shows elevated rates of model-favored words, consistent mostly with grammar and style editing — not that a machine wrote the paper.
How can researchers tell if a paper was written with AI?
The authors count words that language models overuse (such as delve, underscore, pivotal) relative to paper length, and compare against the rate expected without models — calibrated on pre-2020 papers and on 392 papers whose authors declared model use. They show that simpler 'does the word appear anywhere' checks fail on full texts, because longer papers more easily contain any given word by chance.
Is using ChatGPT for scientific writing academic misconduct?
The authors argue no: almost all detected use matches the grammar, clarity and language editing that journal policies already permit, the practice is now the majority norm, and only 0.81% of 2025 papers disclose it. They advocate destigmatizing disclosure rather than enforcement, noting the statistical trace is fading (excess down from 3.5× to 1.5× between 2023 and 2026).
Why do different studies give such different AI-writing percentages?
Methods differ. This paper's per-token count model gives 54% for astronomy full texts in 2025, while presence-based estimators overestimate on full texts (neutral control words alone imply ~51% apparent prevalence) because papers have grown longer; abstract-only studies have far fewer tokens per paper. Measuring counts per token and calibrating each corpus separately reconciles much of the apparent disagreement.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels survive only as captions plus raw axis-tick dumps in the extraction; quantitative figure reading relies on caption text and in-text numbers, both of which are present.
- The metadata abstract supplied with the paper is cut mid-word, but the complete abstract is present at the top of the extracted full text, so nothing in this review depends on the truncated copy.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.