Model Retirement Creates Reproducibility Risk in Biomedical AI Publications

Summary

This bibliometric study quantifies how often biomedical papers depend on large language models whose retirement undermines computational reproducibility. The authors searched PubMed (January 2022 through mid-2026) for original research applying a named LLM to a biomedical task, screened 65,955 abstracts with a GPT-4o-based extraction agent (temperature 0), normalized model names, and excluded encoder-only, protein, and generic family-level mentions. After restricting to the 50 most-used models with vendor-documented lifecycles, 8,931 paper-model mentions across 5,242 publications remained. Human review of a 50-paper subset found 96.2% extraction accuracy, with most errors missed mentions rather than hallucinations. Commercial closed-weight models accounted for 77.7% of mentions (OpenAI 65.7%; GPT-4, GPT-4o, and GPT-3.5-turbo led). Overall, 41.6% of mentions involved a model already retired at publication (2.9%) or scheduled to retire within two years; the median publication-to-retirement interval was 538 days. The authors recommend mandatory exact version-string reporting, open-weight models plus archived weights for long-horizon tasks, and structured publisher metadata. Limitations include abstract-only extraction, a very small validation subset (50 papers) with an undefined accuracy metric, purely descriptive statistics with no confidence intervals or sensitivity analyses, several internal numeric inconsistencies (e.g., 61,077 vs 65,955 abstracts; GPT-5 cited in 182 papers in text vs 170 in Table 1), and code available only on request — an ironic gap given the paper's thesis.