Peerify: Benchmarking Peer-Review Claim Verification
Alireza Daghighfarsoodeh, Sajad Ebrahimi, Ali Ghorbanpour, Soroush Sadeghian, Radin Cheraghi, Negar Arabzadeh, Ebrahim Bagheri
cs.CL · cs.DL
Quick verdict
Peerify benchmarks review-claim verification vs manuscripts (800 claims, 6 variants); supervision well-audited but LLM-judge circularity and untested gains.
Summary
Peerify formalizes manuscript-grounded verification of peer-review claims: decompose a review into atomic claims, retrieve manuscript passages, and label each claim Supported / Partially Supported / Not Supported / Not Determined. The authors build an 800-claim benchmark from NeurIPS 2024 and ICLR 2024 OpenReview artifacts in six variants (PaperSourced, RebuttalSourced, LLMJudged, HighAgreement, verifiable-only, and a 300-claim Human-Verified subset covering 37.5%), with labels generated by GPT-5-mini and Claude-Sonnet-4-6 plus human arbitration on roughly 40% of claims. A human audit reports 90.3% exact agreement between automated and human consensus labels (kappa=0.87; 97.0% under one-step tolerance). Across five LLM verifiers, best accuracy reaches 0.904 on PaperSourced (o4-mini, Dense+Reranker) but only about 0.32 on RebuttalSourced and 0.514 on LLMJudged; no verifier exceeds 0.38 on Human-Verified. Zero-shot MNLI models stay below 0.24 macro-F1 across all 72 configurations. Error analysis attributes 91.3% of PaperSourced errors to over-strictness or retrieval misses (Recall@10 below 0.55), not faulty reasoning. Main limitations: much of the supervision is LLM-generated and the evaluated GPT-5-mini also served as a labeler, raising circularity concerns; RebuttalSourced labels reflect author responses rather than manuscript truth; no significance tests or confidence intervals are reported; and defects like a broken Table ?? cross-reference and a 150-versus-300 instance inconsistency in the ethics statement remain.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Manuscript-grounded peer-review claim verification is a distinct intra-document task requiring long-context, four-way reasoning that three-way entailment cannot express.
- Two-LLM automated supervision with human arbitration is reliable enough to build an 800-claim benchmark: 90.3% exact agreement (kappa=0.87) with human consensus on a 300-claim audited subset.
- Retrieval is necessary but not sufficient: reranked retrieval lifts PaperSourced accuracy to 0.904 (Oracle 0.988), while removing retrieval entirely (full-context) does not lift accuracy on interpretive splits, where no verifier exceeds 0.38 on Human-Verified.
- Off-the-shelf MNLI entailment is not a viable substitute (macro-F1 below 0.24 across all 72 model-retriever-benchmark configurations) due to the label-space and premise-truncation mismatch.
- Calibration matters as much as reasoning: models that over-predict Partially Supported or Supported can look better or worse depending on split composition, so macro-F1 must accompany accuracy.
Strengths
Task formalization with a six-variant benchmark design
Section 3.4 separates label-confidence (PaperSourced all-Supported sanity split, HighAgreement slice where two disjoint-evidence signals agree, verifiable-only factual filter) from task difficulty (RebuttalSourced/LLMJudged), letting readers grade conclusions by supervision reliability — a design most LLM-labeled benchmarks lack. Prompts are reproduced verbatim (Appendix C, Figures 3-5) and code/data/labels are stated to be released.
Unusually deep supervision auditing
A.8/A.9: human inter-annotator kappa ≈ 0.96; automated-vs-human 271/300 = 90.3% exact (kappa = 0.87), 97.0% one-step tolerant, 1.0% hard polarity flips; Table 7's confusion matrix shows disagreements clustering at the Partially Supported boundary; Table 8 shows reliability flat across objective (90.5%) vs subjective (90.1%) claims — the exact check an annotation-bias account would fail.
Non-LLM baselines are fair, not strawmen
Honest, mechanistic error analysis and calibration accounting
Measured operating cost supports deployment claims
Weaknesses
LLM-judge circularity between labelers and scored verifiers
RebuttalSourced supervision measures author responses, not manuscript truth
No inferential statistics anywhere
Benchmark composition depends on a low-recall extractor
Internal inconsistencies and production defects
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Peerify (arXiv 2609.25046) targets a task the authors argue is underexplored: verifying whether claims made by peer reviewers are grounded in the manuscript under review, as opposed to FEVER/SciFact-style open-corpus verification (Sections 1, 2.1). The deliverables are a three-stage pipeline — Qwen3-4B claim extraction, BM25/dense retrieval (optionally cross-encoder reranked) over Docling-chunked manuscripts, and a four-way groundedness verifier (Supported / Partially Supported / Not Supported / Not Determined) — plus an 800-claim benchmark from NeurIPS 2024 and ICLR 2024 OpenReview artifacts in six variants (Table 1). Central claims: (i) the task needs long-context, four-way reasoning beyond three-way entailment; (ii) automated two-LLM supervision with human arbitration is reliable (90.3% exact, kappa=0.87 vs human consensus on 300 audited claims); (iii) retrieval is necessary but not sufficient (Section 5.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Peerify?
Peerify is a pipeline and benchmark for checking whether statements made in peer reviews are actually supported by the reviewed paper. It splits a review comment into atomic claims, retrieves relevant manuscript passages, and labels each claim as Supported, Partially Supported, Not Supported, or Not Determined, together with the evidence used. The benchmark has 800 claims from NeurIPS 2024 and ICLR 2024 public reviews.
How reliable are Peerify's labels?
A 300-claim subset (37.5% of the benchmark) was hand-labeled by two annotators who agreed at kappa ≈ 0.96. The automated labels (from GPT-5-mini and Claude-Sonnet-4-6 with human arbitration) match the human consensus on 90.3% of claims exactly (kappa = 0.87), rising to 97.0% if a one-step label difference is tolerated; hard flips between supported and unsupported occur in 1.0% of cases.
How well do LLMs perform at verifying peer-review claims?
It depends on claim provenance: on claims copied from the paper itself, the best model reaches 0.904 accuracy; on claims from real author-reviewer disputes, the best is only about 0.32; on the hand-labeled subset no model exceeds 0.38. Zero-shot entailment models (RoBERTa/BART/DeBERTa-MNLI) stay below 0.24 macro-F1 in all 72 tested configurations.
Is the Peerify code and data available?
The paper states that the pipeline, benchmark, labels, prompts, and code are released at github.com/Reviewerly-Inc/Peerify, built from publicly available OpenReview data, with every prompt reproduced verbatim in the paper's appendix.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 1's venue-level rows (NeurIPS/ICLR paper counts 4236/5778 and their percentage splits) are column-scrambled in the PDF extraction; the per-variant rows parse coherently and were used instead.
- Figures 1-2 are embedded only as partial axis/legend text in the extraction; exact full-context and label-distribution values were therefore taken from Tables 14 and the Figure 2 caption text rather than read off the plots. No end-of-document truncation was observed (Appendix E closes with a complete sentence).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.