AppliedScientist: Automated Scientific Revision Through Iterative AI Reviewing
Vidushee Vats, Karun Sharma, Shengzhi Li, Shichao Pei
cs.AI
Quick verdict
Closed-loop AI reviewer-guided revision of real rejected ICLR papers fixes 85.3% of execution flaws, not novelty; promising, but small-n and evaluator-circular.
Summary
AppliedScientist couples an autonomous AI scientist with an AI reviewer in a closed loop: the scientist revises rejected ICLR papers' code, experiments, and manuscripts across five rounds, while a memoryless reviewer independently scores each version and supplies the next round's feedback. Across 30 papers (25 rejected, 5 borderline) spanning five domains and three guidance conditions (human-review-initialized, AI-initialized, fixed-prompt self-revision), human-initialized runs improve from 5.37 to 6.53 under the authors' reviewer, and Stanford Reviewer independently scores the same trajectories rising from 5.50 to 6.15 (best-revision aggregation; Stanford scores no other condition). Their reviewer benchmarked on 650 ICLR papers reaches ρ=0.571 with mean human ratings and 76.2% venue-decision agreement. The system resolves 128 of 150 execution weaknesses (85.3%) but only 2 of 18 idea/novelty weaknesses (11.1%), at ~$65 per paper. Limitations: no statistical testing with n=30, self-revision baseline never independently evaluated, and the novelty finding is partly built in by design—the scientist is instructed to keep the central contribution fixed. No code or data release is mentioned.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reviewer-guided iterative revision of real rejected ICLR papers improves papers substantially more than autonomous self-revision with a fixed prompt, with the scientist's capabilities held identical across conditions.
- Gains generalize across five research domains and are corroborated by the independent Stanford Reviewer on the human-initialized condition (5.50→6.15 at best revision).
- AppliedScientist resolves 128 of 150 execution-related weaknesses (85.3%) but only 2 of 18 idea-related weaknesses (11.1%), indicating iterative revision strengthens evidence and presentation but not novelty or significance.
- The authors' structured AI reviewer outperforms existing automated reviewers on most dimensions of a 650-paper ICLR benchmark (ρ=0.571 with mean human ratings; 76.2% venue-decision agreement).
- Unresolved novelty objections are genuine: all 23 cited prior works behind the 17 surviving objections were verified as real and relevant (Appendix B.4).
Strengths
Real execution, not text-only rewriting
Unlike Apres, ARIES, and XtraGPT, revisions run code and execute new experiments on 25 rejected + 5 borderline ICLR papers with public, single-GPU-reproducible repositories (Table 1), making the execution-resolution claims (85.3%) empirically checkable in principle.
Unusually candid failure analysis
The 2/18 idea-resolution rate is reported as a headline result, and Appendix B.4 manually verifies all 23 citations behind the 17 surviving novelty objections against the arXiv API — showing the unresolved objections rest on real prior work, not hallucinated comparisons.
Clean isolation of the guidance signal
Reviewer validated before deployment as reward signal
Weaknesses
No statistical inference; favorable aggregation
Self-revision baseline never externally judged
The 11.1% idea-resolution result is partly built in
Inconsistent numbers across sections
Scientist component underspecified; no artifacts released
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AppliedScientist (arXiv 2609.14738v2, UMass Boston) studies whether the review→revise cycle of scientific publishing can be automated end-to-end, including code and experiments. A scientist agent revises real rejected ICLR papers over up to five rounds; a memoryless AI reviewer independently scores each version and feeds the next round. Three conditions isolate the guidance signal: human-initialized (venue reviews for V1), AI-initialized (the authors' reviewer from V1), and autonomous self-revision with a fixed prompt (Table 2). The headline claims: guided revision beats self-revision; gains hold under an external judge; and revision resolves execution weaknesses (128/150, 85.3%) far more often than idea/novelty weaknesses (2/18, 11.1%).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AppliedScientist?
A closed-loop research system from UMass Boston in which an autonomous AI scientist iteratively revises a rejected paper — editing its code, running new experiments, and rewriting the manuscript — while an AI reviewer with no memory of earlier rounds independently reviews each new version and supplies the feedback for the next round. The central research contribution is deliberately kept fixed.
What did AppliedScientist achieve in its experiments?
On 30 real ICLR papers (25 rejected, 5 borderline) across five domains, papers revised with reviewer feedback improved far more than papers revised by the scientist alone with a fixed prompt. Under the authors' reviewer, the average score rose from 5.37 to 6.53 at each paper's best revision, and the independent Stanford Reviewer scored the same trajectory rising from 5.50 to 6.15.
Can AI revise a rejected paper well enough to get it accepted?
This paper suggests only partially: AI revision reliably fixed execution problems — missing baselines, weak evidence, unclear writing — resolving 85.3% of them, but it resolved only 11.1% of concerns about the idea's novelty or significance. The system is also instructed to keep the paper's core contribution unchanged, so fundamental repositioning is out of scope by design.
How much does it cost to run AppliedScientist on one paper?
About $65 and roughly 9 hours on a single 96 GB GPU (RTX Pro 6000) for five revision rounds including experiment execution. The full 30-paper evaluation cost about $2,000 and 270 GPU-hours; the authors cite this budget as the main limit on evaluation size.
Is the code or data for AppliedScientist available?
The paper references a project page (theappliedscientist.github.io) but does not state that the system code, prompts beyond those printed in the appendix, or the revision trajectories are released. The 30 evaluated papers were selected for having public, single-GPU-reproducible repositories.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-10 are axis-label text only; curve shapes, error bars, and exact trajectories could not be analyzed.
- Table 7 and the trajectory columns appear misaligned (Vmax followed by a seventh value; Stanford deltas sometimes exceed the column span), likely a PDF-extraction artifact — apparent numeric inconsistencies may partly stem from this.
- Supplementary sections reference prompts and tables (C.3/C.4, Table 8) that appear truncated in the extraction.
- arXiv v2 (19 Sep 2026); the paper cites 2026-dated systems (ScholarPeer, DeepReviewer 2.0, APRES), so the reviewer-benchmark landscape may already have moved.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.