Positioning manuscripts in the scientific landscape with agentic AI
Jiawen Chen, Zichen Zhang, Bingxuan Li, Quan Sun, Yiyan Zhang, Edric Tam, Jinjie Lin, Didong Li, et al.
cs.AI
Quick verdict
Journal-landing prediction via agentic AI: 50.3% Top-1 vs 41.2% prompted GPT-5 on 2,184 preprints; convincing benchmark, but leakage audit is self-report.
Summary
The paper introduces PASS, a three-agent LLM system (manuscript understanding, literature retrieval, journal-fit reasoning) built on GPT-5 that predicts where a biomedical manuscript will ultimately be published and returns quality scores, rationales, and improvement feedback. The benchmark comprises 2,184 bioRxiv/medRxiv preprints (v1 on or after 1 Oct 2024, published by 19 Apr 2026) across 16 SCImago-mapped fields, with candidate sets of the top-100 SJR journals per field plus multidisciplinary venues. PASS achieved 50.3% Top-1 and 86.1% Top-5 accuracy versus 41.2%/79.1% for a task-engineered GPT-5 and 32.8%/77.6% for naive prompting, and outperformed JANE and a retrained Pubmender. Ablations attribute 29.2% of PASS's gain over the engineered baseline to literature retrieval, 22.5% to manuscript signals, and 10.1% to instructions. PASS kept near-full accuracy from the abstract alone (53.8%/85.9% vs 53.0%/86.8% full text), while baselines degraded without full text. Score associations with SJR/CiteScore/H-index use Wilcoxon tests, but accuracy differences lack confidence intervals, and some cross-section figures (50.3% vs 53.0%) go unexplained. Limitations: evaluation conditions on papers that landed inside the candidate journal sets; the leakage audit relies on GPT-5 self-report; the author venue-history module is not ablated; the human evaluation is small and partly self-assessed; system code is not stated as released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PASS predicts the eventual publication journal of biomedical preprints with 50.3% Top-1 and 86.1% Top-5 accuracy across 2,184 manuscripts in 16 field-specific candidate spaces, exceeding task-engineered GPT-5 (41.2%/79.1%), naive GPT-5 (32.8%/77.6%), JANE, and a retrained Pubmender.
- Literature retrieval is the largest single contributor to PASS's advantage, accounting for 29.2% of the performance gain over the task-engineered GPT-5 baseline, with closely related studies (14.6%) contributing more than review/benchmark papers (7.3%).
- Journal-landing signal is concentrated in abstract, introduction, and discussion: PASS retains near-full accuracy from the abstract alone while both GPT-5 baselines degrade without full text.
- PASS's manuscript-level scores (impact potential, novelty, conceptual soundness, writing quality) align monotonically with independent journal-tier metrics (SJR, CiteScore, journal H-index).
- An independent six-reviewer human evaluation found 88% agreement with recommended journal tier, 80.2% of improvement comments judged useful, and 100% agreement that PASS correctly understood the manuscript.
Strengths
First formal benchmark for journal-landing prediction with genuine temporal controls
2,184 bioRxiv/medRxiv preprints restricted to v1 dates on/after 1 Oct 2024, frozen 19 Apr 2026 (§4.3), with retrieved literature date-filtered to pre-submission records (§4.1); benchmark metadata released on Hugging Face.
Transparent, quantified component attribution
Leave-one-out ablations express each component as a fraction of the PASS–task-engineered gap: literature 29.2%, manuscript signals 22.5%, instructions 10.1% (§2.4), further decomposed into related studies (14.6%) vs reviews/benchmarks (7.3%) and stratified by paper type (method 11.73 vs resource 7.33 mean retrieved papers).
Conservative baseline configuration
External validation of the interpretability claims
Fairer-than-usual legacy-tool comparison
Weaknesses
Leakage audit is a self-report
Author venue-history module is never ablated
Evaluation conditions on the outcome
Thin statistical reporting and unexplained numeric drift
Small, self-selected human evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether agentic LLM systems can solve a problem adjacent to peer review: predicting where a manuscript will land. PASS (§2.1) chains three GPT-5-driven agents — manuscript understanding (structured profile, contribution type, query seeds), literature retrieval (PubMed/Europe PMC/Semantic Scholar queries, date-filtered to pre-submission, embedding pre-rank + LLM re-rank, evidence split into closely related vs review/benchmark papers, plus author venue-history summaries), and journal-fit reasoning (joint 0–10 scoring of novelty/impact/soundness/writing plus five ranked journals from a field-specific candidate list). Three claims carry the paper: (1) journal landing is a predictable, benchmarkable endpoint (50.3% Top-1 / 86.1% Top-5 on 2,184 preprints); (2) literature retrieval, not prompting, drives the gain (29.2% of the PASS–baseline gap); (3) the signal lives in narrative sections, so abstract-only input nearly matches full text. A released platform (ratemypaper.ai) extends the benchmark into a product with quality scores and revision roadmaps.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PASS and what does it do?
PASS is an agentic AI system built on GPT-5 that reads a biomedical manuscript, retrieves related literature from PubMed, Europe PMC and Semantic Scholar, and returns five recommended journals with rationales, quality scores (novelty, impact potential, conceptual soundness, writing quality), and manuscript-improvement suggestions. A public version is available at ratemypaper.ai.
How accurate is PASS at predicting where a paper will be published?
On 2,184 bioRxiv/medRxiv preprints across 16 biomedical fields, PASS placed the eventual publication journal first 50.3% of the time and within its five suggestions 86.1% of the time, versus 41.2%/79.1% for a carefully prompted GPT-5 without retrieval and 32.8%/77.6% for naive prompting. Note the task is limited to candidate sets of roughly the top-100 journals per field.
Could the model just have memorized which papers went to which journals?
The authors reduce this risk by using only preprints first submitted after October 2024, filtering retrieved literature to pre-submission dates, and running an audit in which GPT-5 said it could not identify the destination journal for any of the 2,184 papers. However, that audit depends on the model's own yes/no self-report, which is a known weakness, so some residual memorization risk cannot be excluded.
Do you need the full paper for a good journal prediction?
No. PASS ran its main evaluation using only the abstract, introduction, and discussion, and scored 53.8% Top-1 with the abstract alone versus 53.0% with the full manuscript, while the GPT-5 baselines lost several points without full text. The authors argue venue-relevant signal is concentrated in these narrative sections.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Supplementary Figures S1-S4 and the numeric per-panel values behind Fig 2/Fig 5 are not included in the provided text, so field-level accuracy claims could not be checked individually.
- Exact agent prompts are described but not quoted; the precise leakage-control wording and candidate-table format could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.