PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress
Kevin Qinghong Lin, Siyuan Hu, Pan Lu, Yu Chen, Yanzhe Chen, Owen Queen, Yupeng Chen, Jialin Yu, et al.
cs.CL · cs.MA
Quick verdict
Three-level agent for evidence-grounded pre-submission feedback; promising 70.6% author acceptance, but no baseline or independent correctness validation.
Summary
PaperDoctor is an agent framework that reframes automated paper assessment as pre-submission 'diagnosis' rather than accept/reject judging. Given a manuscript plus its code and data, it parses the paper (Mathpix markdown, rendered page images, tree-sitter-indexed code, bibliography) and runs a three-level pipeline: L1 surface screening (writing, figures via VLM, citation verification via web search, dense claim extraction); L2 typed verifiers routing each claim to code audit, step-by-step theory re-derivation, literature/novelty checks, or experiment-design review; and L3 selective experiment reproduction under priority ranking and author approval. Every finding is emitted as a triple of observation, evidence pointer, and revision suggestion. On 30 in-progress papers from 25 graduate students, authors accepted 70.6% of 1,299 items and rated all 30 reviews positive (mean +1.30 on a -1 to +2 scale). Against referee reports and the Stanford Agentic Reviewer on 40 papers from four sources (Agents4Science, ICML orals/spotlights, Nature Communications, Nature Human Behaviour), PaperDoctor provides evidence in 100% of findings by design versus 45.2% (referees) and 2.2% (agentic reviewer), and covers external literature and code more. In L3, only 18.0% of reproduction plans are executable initially; among executed plans, mismatch and failure rates are high (e.g., only 14.4% of executed plans on ICML papers matched reported numbers). Limitations: no baseline arm in the human study, author self-report as the sole correctness measure, LLM-parsed comparisons, and no component ablations or cost analysis.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A three-level hierarchical pipeline (L1 surface screening, L2 typed claim verifiers, L3 approval-gated reproduction) delivers advisor-style, evidence-grounded feedback on in-progress papers at scale.
- Every PaperDoctor finding carries both quoted evidence and a revision suggestion by design - 100% versus 45.2% (evidence) for human referee reports and 2.2% for the Stanford Agentic Reviewer on 40 papers.
- Authors accepted 70.6% of 1,299 item-level findings across 30 in-progress papers, and all 30 holistic ratings were positive (mean +1.30 on a -1 to +2 scale).
- Selective experimental reproduction surfaces claim-number divergences invisible to reading-only review: only 14.4% of executed plans on ICML papers matched reported numbers, while 72.2% of the few executable Agent4Science plans did.
Strengths
Auditability by construction, quantified against baselines
The (observation, evidence, suggestion) contract is enforced on every output; Fig 3B shows 100% of PaperDoctor findings carry evidence+specification versus 45.2% (evidence) for referee reports and 2.2% for the Stanford Agentic Reviewer, with suggestions in 35.9% vs 1.5% joint coverage.
Cost-tiered, safety-gated pipeline design
Sec 2.2-2.4 organizes checks by verification cost: parallel L1 surface skills, typed L2 claim routing (code audit, step-by-step theory re-derivation, search-grounded novelty checks, experiment-design review emitting a declarative reproduction plan), and L3 execution only after author approval with estimated GPU hours, dataset and storage footprint (Sec 2.4 Manual Approval).
Largest item-level author evaluation in this space, honestly reported
Reproduction findings are substantive and independently checkable
Coverage extends beyond the manuscript
Weaknesses
No baseline arm or independent adjudication in the human study
Confounded comparison design in Fig 3
L3 verdicts are contextual LLM judgments with low yield
Flagship capability excluded from the author study; no ablations
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PaperDoctor (arXiv 2609.16995, cs.CL) reframes automated paper assessment as pre-submission diagnosis rather than accept/reject judging. Given a draft plus its code and data, the system emits findings, each required to be a triple: an observation (Why), an evidence pointer to a sentence, equation, code line, or URL (Where), and a revision suggestion (How). Sec 1 motivates this via the gap between advisor-style feedback and judge-style reviewers; Sec 4 and Table 1 position the system against autoresearch writers (AI Scientist), review generators (MARG, Reviewer2, DeepReview, TreeReview), and verification tools (CiteAudit, PaperBench, AutoReproduce), claiming a unique combination of report-style output, grounded evidence, revision suggestions, visual and code coverage, and reproduction for in-progress papers. The central empirical claims: (i) 70.6% author acceptance across 1,299 items on 30 in-progress papers with all-positive holistic scores; (ii) 100% evidence grounding by construction versus 45.2% (referees) and 2.2% (agentic reviewer) on a 40-paper, four-source benchmark; (iii) reproduction surfacing claim-number divergences invisible to reading-only review.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PaperDoctor?
PaperDoctor is an AI agent framework that reviews scientific papers before submission. Instead of returning an accept/reject verdict like a reviewer, it acts like an advisor marking up a draft: it flags issues in writing, figures, citations, theory, code, related work, and experiment design, and every finding must quote the exact evidence (a sentence, equation, code line, or URL) and include a concrete revision suggestion.
How well does PaperDoctor perform according to the paper?
In a study of 30 in-progress papers by 25 graduate students, authors accepted 70.6% of the 1,299 individual findings and rated all 30 overall reviews positively (mean +1.30 on a -1 to +2 scale). Compared on 40 published papers, 100% of PaperDoctor's findings carry quoted evidence versus 45.2% for human referee reports and 2.2% for the Stanford Agentic Reviewer. Note these acceptance numbers are the authors' own self-reports, not independent correctness checks.
Can PaperDoctor rerun a paper's experiments?
Yes, selectively and only with the author's approval. It builds a prioritized reproduction plan with estimated GPU hours, dataset size, and storage, then executes after sign-off. In the paper's tests only 18% of plans could run immediately (most were blocked by missing code or instructions), and results often diverged: only 14.4% of executed plans on ICML papers matched the reported numbers, though training-based reruns fared worst (11.3% pass).
How is PaperDoctor different from AI peer reviewers?
AI peer-review systems are optimized to judge finished papers for venues - scores and verdicts. PaperDoctor targets drafts still in progress and serves the author: it never emits a critique without a grounded location in the paper and a proposed fix, it separates confident 'errors' from uncertain 'warnings', and it refuses to spend real compute on rerunning experiments without explicit author approval.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract is truncated and Figures 1-5 are not visually analyzable; numeric readings for Figs 2-4 were recovered from text embedded in the figure dumps and may contain extraction noise.
- Supplementary implementation details (prompts, model versions per skill, cost accounting) are not present in the extracted text, limiting methodology verification.
- Some figure-embedded statistics (e.g., per-source plan counts in Fig 4D) appear garbled in extraction; cross-checked against main-text numbers where possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.