More Than Mimicking Reviewers: Evaluating LLMs for Pre-Submission Peer Review
cs.AI
Quick verdict
LLM pre-submission review reaches ~85% panel-issue coverage at 5× cost but compresses poorly; sound protocol, yet ten papers and a shaky judge limit it.
Summary
The paper asks whether an LLM system can give authors peer-review-style feedback before submission. The authors build a leakage-controlled benchmark from 10,000 ICLR 2026 OpenReview submissions, recovering 3,398 manuscripts in reviewer-visible versions (arXiv pre-submission versions and timestamped OpenReview PDFs), with frozen train/validation/test splits (2,446/272/680) keyed by duplicate-family IDs. Their pipeline generates hundreds of atomic, evidence-grounded concerns, then compresses them into a 32-item report. On a ten-paper diagnostic with 178 manually corrected historical issues, independent sampling covers 44.9% of issues (49.5% seriousness-weighted); a deduplication-and-refill pipeline raises this to 78.7% strict and 84.9% weighted, but at 3.6× the requests and 5.2× the tokens (96.61M tokens for ten papers). Under a hidden Top-32 oracle, 32 slots preserve the full 256-candidate pool's 79.3% weighted coverage, yet paper-only selectors retain only 40–44%; ablations attribute about 6 points to clustering and about 35 to representative selection. The authors also show the matching judge is fragile (two matchers mark 660 vs 8 of 45,568 pairs, κ=.015) and that same-family preference can confound selection. Main limitations: headline numbers rest on ten papers with no variance estimates; the 0.70 judge threshold is unvalidated by humans; unmatched concerns' validity remains unaudited; and no direct 'ask for 32 concerns' baseline is reported.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A deduplication-and-refill generation pipeline raises ten-paper historical coverage from 44.9% to 78.7% strict (49.5%→84.9% seriousness-weighted), at 3.6× requests and 5.2× tokens.
- Thirty-two oracle-selected slots preserve the full 256-candidate pool's 79.3% weighted coverage, so report capacity is not the compression bottleneck.
- Paper-only selectors lose roughly 35 weighted points versus the cluster oracle, making representative selection — not clustering (~6 points) — the dominant failure mode.
- Matcher miscalibration and same-family preference are first-order validity threats: two plausible judges mark 660 vs 8 of 45,568 pairs (κ=.015), and a Sol selector chooses 319/320 Sol candidates from mixed pools.
- Historical-review agreement and independent validity must be evaluated separately because unmatched AI concerns may be concrete issues the human panel missed.
Strengths
Leakage-controlled benchmark construction
Pre-review versions are recovered via two independent paths (arXiv mapping: 2,123; OpenReview timestamps: 1,548; 273 overlap → strict cohort 3,398, Fig. 1); generation is frozen before reviews open, and splits (2,446/272/680, Table 1) are keyed by duplicate-family IDs to block resubmission leakage.
Honest cost–coverage accounting
Table 2 and §4.4.2 report the full price of coverage: 3.6× requests (525→1,905), 5.2× tokens (18.66M→96.61M), and diminishing returns (256→512 candidates adds 55.61M tokens for nine issues); Table 5 shows adaptive stopping cuts 92.3% of tokens but drops weighted coverage .849→.660.
Self-critical failure analysis
Clean decomposition of the compression gap
Agreement-vs-validity distinction
Weaknesses
Headline claims rest on ten papers
All metrics flow through one unvalidated LLM judge
Missing direct-generation baseline
Title claim (validity beyond imitation) is unmeasured
Deployable system quality and cost
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper evaluates an author-facing LLM 'pre-submission review' system on ICLR 2026 manuscripts: generate a broad pool of atomic, evidence-grounded concerns (up to 512 per paper), then compress to a 32-item report. Three claims anchor the abstract: (1) dedup-and-refill generation lifts historical-issue coverage from 44.9% to 78.7% strict and 49.5% to 84.9% seriousness-weighted on a ten-paper diagnostic (Table 2), at 3.6× requests and 5.2× tokens; (2) a hidden Top-32 Oracle retains the full 256-candidate pool's 79.3% weighted coverage while paper-only selectors keep only 40–44% (Table 3); (3) ablations locate the loss in representative selection, not clustering. A fourth, softer claim — that unmatched AI concerns may be valid beyond imitation — is explicitly left unmeasured (§4.5.1, Appendix B).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper show about LLMs doing peer review before submission?
It builds a leakage-controlled benchmark of 3,398 ICLR 2026 manuscripts whose pre-review versions are recoverable, and evaluates an LLM system that generates hundreds of atomic concerns about a draft and compresses them into a 32-item report. On a ten-paper diagnostic, broad generation with dedup-and-refill recovers 78.7% of the issues raised by the historical human review panel (84.9% when weighted by seriousness) — but paper-only compression to 32 items loses roughly half of that coverage.
What are the main limitations of the study?
Headline numbers come from only ten papers with 178 issues and no uncertainty estimates; all coverage is measured by a single LLM judge at a 0.70 threshold the paper itself shows is fragile (a second judge marked 8 positive pairs where the first marked 660 of 45,568, κ=.015); the validity of AI-only concerns that match no human review is asserted but not yet audited by experts; and no simple 'directly ask for 32 concerns' baseline is reported.
How much does this pre-submission review approach cost?
The dedup-and-refill pipeline needed 1,905 requests and 96.61M tokens for ten papers — 3.6× the requests and 5.2× the tokens of plain independent sampling. Returns diminish sharply: expanding from 256 to 512 candidates per paper added 55.61M tokens for only nine additional issues.
Did the LLM find issues that human reviewers missed?
The paper presents two concrete, paper-grounded examples of specific unmatched concerns, but explicitly states that expert adjudication of whether such AI-only concerns are valid remains incomplete — so this claim, central to the title, is not yet established by evidence in the current version.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure images (Figs. 1–3) appear in the extraction only as captions and axis-label fragments; plotted per-paper detail beyond the labeled numbers was not recoverable from text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.