PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews
cs.SE · cs.AI · cs.CL
Quick verdict
Analyzes 888 review-automation papers to ground the PRISMA-LLM reporting checklist; evidence-rich and candid, though descriptive and not yet validated.
Summary
The paper analyzes SciLitBench, a corpus of 888 review-automation papers with 14,726 annotations, to characterize how AI has entered systematic reviewing and what that implies for reporting. Since 2023, 38.0% of software/product papers reported no evaluation versus 9.3% of LLM papers; mean reporting richness (a 0–15 descriptive index) rose with LLM workflow complexity, from 2.6 for software-only and 5.2 for prompt-only to 8.4 for engineered and 9.1 for retrieval/adapted/agentic workflows. Publication volume grew roughly 4.7% monthly from 2020 onward, LLM use in life sciences and medicine approached 60% in recent windows, and 84.1% of identifiable model use was proprietary. Strikingly, 52% of 118 LLM papers with purely positive evaluations still reported an unmet reliability or performance bar, separating usefulness from adequacy for delegation. From these patterns the authors derive PRISMA-LLM: a 16-item universal checklist, five implementation-disclosure levels from off-the-shelf tools to multi-agent systems, and consequence-sensitive evaluation expectations. Limitations, largely self-acknowledged, include a biomedical-leaning corpus, annotations produced by Llama 3.3-70B with human adjudication, a complexity–reporting gradient confounded by study purpose and venue, an unvalidated richness index, and no Delphi consensus or user testing; core results rest on a single self-built corpus, with materials hosted on GitHub.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Since 2023, software/product papers report markedly less evaluation than LLM papers (38.0% vs 9.3% with no evaluation; mean richness 3.3 vs 6.3).
- Paper-level reporting coverage increases monotonically with LLM workflow complexity (2.6 for software-only, 5.2 prompt-only, 8.4 engineered, 9.1 retrieval/adapted/agentic since 2023).
- Positive overall impressions frequently coexist with unmet reliability requirements: 52% of 118 positive-only LLM papers report an unmet high-bar concern.
- A three-layer framework — 16-item checklist, five implementation-disclosure levels, consequence-sensitive evaluation — can make AI-assisted reviews auditable while separating implementation complexity from methodological risk.
- LLM model access remains predominantly proprietary (84.1%), with no statistically significant differences in openness across domains.
Strengths
Unusually large empirical base for a guideline paper
888 full-text papers and 14,726 annotation items (Table S1) ground the framework; headline contrasts are quantified (38.0% of software/product papers vs 9.3% of LLM papers with no evaluation since 2023; richness 3.3 vs 6.3 vs 6.0).
Monotone complexity–reporting gradient with stage-robustness checks
Figure 3b shows mean richness rising 2.6 → 5.2 → 8.4 → 9.1 and no-evaluation falling 44.4% → 13.4% → 1.7% → 0% across software-only, prompt-only, engineered, and retrieval/adapted/agentic groups; Supplementary S4 shows the software gap persists within every review stage, ruling out stage-composition as the sole explanation.
The usefulness–adequacy distinction is a genuinely novel, actionable finding
Disciplined epistemic positioning
Careful inferential hygiene where it counts
Weaknesses
Bespoke, unvalidated measurement index carries the headline claims
Annotation pipeline quality is deferred, not demonstrated
Complexity gradient is confounded and unadjusted
Exploratory precision/recall inventory invites over-reading
Framework usability untested; growth narrative slightly flattering
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
PRISMA-LLM: Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PRISMA-LLM?
PRISMA-LLM is a proposed (not yet officially endorsed) reporting framework for systematic reviews that use LLMs or AI-enabled software. It has three layers: a 16-item universal checklist, five implementation-disclosure levels (off-the-shelf software, prompt-only, structured/engineered, retrieval/adapted/single-agent, multi-agent), and consequence-sensitive evaluation expectations that rise when AI outputs can change the evidence base.
What did the study find about evaluation reporting in review-automation papers?
Analyzing 888 papers, it found that since 2023, 38.0% of software/product papers reported no evaluation versus 9.3% of LLM papers, and that reported evaluation detail increases with LLM workflow complexity: mean scores of 2.6 (software-only), 5.2 (prompt-only), 8.4 (engineered), and 9.1 (retrieval/adapted/agentic) on a 0–15 index.
Does a positive evaluation mean an LLM review workflow is reliable enough to trust unsupervised?
Not according to the authors' own reports: 52% of 118 papers with purely positive evaluations still reported at least one limitation stating the workflow remained below the reliability or performance bar required for its intended use, showing that 'useful' and 'adequate for delegation' are different claims.
Is PRISMA-LLM an official PRISMA extension?
No. The paper explicitly states it is not an official extension endorsed by the PRISMA Executive; formal Delphi consensus development and prospective user testing are named as future validation steps. The checklist and materials are hosted on GitHub.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures (Figs 1–5, S1–S6) appear in the extraction only as captions plus in-text statistics; panel-level visual content (e.g., exact window-by-window shares in Fig 4a) could not be inspected.
- Annotation-quality statistics (inter-annotator agreement, LLM-vs-human accuracy) are deferred to the companion SciLitBench preprint (ref [11]) and are genuinely absent from this text, so annotation reliability could not be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.