Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics
Shakiba Amirshahi, Sajad Ebrahimi, Hai Son Le, Negar Arabzadeh, Ebrahim Bagheri
cs.CL
Quick verdict
Claims 23/29 LLM-judge review metrics conflate style with substance; solid Wilcoxon+TOST design, but nested data and data-derived equivalence bounds temper it.
Summary
This CIKM '26 empirical study tests whether LLM-as-a-judge metrics for peer-review quality measure substantive content or linguistic presentation. The authors score 674 original ICLR/NeurIPS reviews and 4,044 meaning-preserving LLM rewrites of them (PeerPrism dataset; 160 papers, 2021–2024) using 29 content-oriented metrics drawn from four prior frameworks (ReviewEval, REMOR, RottenReviews, ScholarPeer), with each metric's original prompt applied unmodified under GPT-5-mini, replicated with Gemini-2.5-Flash. For each metric, paired score differences are tested with a Bonferroni-corrected Wilcoxon signed-rank test (surface sensitivity) and a TOST equivalence test with bound δ=0.2σ (robustness). Rewrites were human-validated on 50 sampled pairs (mean faithfulness 4.28–4.96/5; 94% judged to give equivalent feedback). Headline results: 23 of 29 metrics shift significantly under rewriting; only six — ReviewEval's Total Insights, Actionable Insights, Actionability Score, and REMOR's Criticism, Materials & Methods, Results & Discussion — meet the robustness criterion. Direction differs by source: RottenReviews scores rise (e.g., Overall Score +5.573 on a 0–100 scale) while human-anchored ScholarPeer scores fall (e.g., Analytical Depth −1.192 on a 1–10 scale). Main limitations: nested data (multiple rewrites per review, reviews per paper) is acknowledged but not modeled, per-metric sample sizes are unstated, the equivalence bound is derived from the same data, and both judges are lightweight-tier models.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- 23 of 29 content-oriented LLM-judge review metrics assign significantly different scores (Bonferroni-corrected Wilcoxon, α≈0.00172) to meaning-preserving LLM rewrites of the same reviews.
- Only six metrics — ReviewEval's Total Insights, Actionable Insights, Actionability Score and REMOR's Criticism, Materials & Methods, Results & Discussion — provide positive TOST evidence of robustness (δ=0.2σ).
- Rewriting shifts scores upward for ReviewEval/REMOR/RottenReviews dimensions (e.g., Overall Score +5.573 on [0,100]) but downward for human-anchored ScholarPeer dimensions (e.g., Analytical Depth −1.192 on [1,10]).
- Classifications are largely stable across GPT-5-mini and Gemini-2.5-Flash (24/29 per §1), indicating the conflation is not specific to one judge model.
- Metrics anchored in concrete, identifiable content are more robust than holistic quality judgments, so broad constructs should be decomposed or validated against meaning-preserving rewrites before deployment.
Strengths
Sound two-criterion measurement framework
Section 2 separates statistical significance (Wilcoxon signed-rank) from practical equivalence (TOST), yielding an explicit four-way classification (sensitive-only, robust-only, sensitive∩robust, inconclusive) and explicitly rejecting the p>0.05-as-no-effect fallacy (§4 Finding 3).
Breadth with prompt fidelity
29 content-oriented metrics from four independent frameworks (ReviewEval, REMOR, RottenReviews, ScholarPeer) are audited with their original prompts and rubrics used 'without modification' (§3), spanning [0,1], [0,5], [0,100], [1,10] and [0,∞) scales.
The key assumption is human-validated
Cross-judge replication and transparent per-metric reporting
Actionable design insight
Weaknesses
Nested non-independence is acknowledged but not remediated
Per-metric sample sizes never stated
Data-derived equivalence bound is partly circular
Judge-tier overreach
ScholarPeer anchoring ambiguity and self-sourced evaluation material
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This CIKM '26 paper (6 pages, DOI 10.1145/3799682.3840033) asks a measurement-validity question rather than proposing a new metric: do LLM-as-a-judge metrics for peer-review quality respond to a review's substance or to its linguistic presentation? Section 2 formalizes two complementary criteria — **surface sensitivity** (does meaning-preserving rewriting shift scores at all, via Wilcoxon signed-rank) and **robustness** (is any shift practically negligible, via TOST equivalence) — applied to 29 content-oriented metrics from four prior frameworks: ReviewEval [11], REMOR [27], RottenReviews [8], ScholarPeer [12]. Test material is the PeerPrism rewritten partition [23]: 4,044 LLM rewrites of 674 human reviews across 160 ICLR/NeurIPS papers (2021–2024). Central claims, anchored in Table 1: under GPT-5-mini judging, 23/29 metrics shift significantly at Bonferroni-corrected α=0.05/29≈0.00172; only six meet the robustness criterion; results broadly replicate under Gemini-2.5-Flash.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'judging a review by its cover' mean in this paper?
It refers to the finding that LLM-based metrics scoring peer-review quality partly react to how a review is written (fluency, organization, polish) rather than only to what it says. When 674 real ICLR/NeurIPS reviews were rewritten by an LLM with the evaluative content kept identical, 23 of 29 evaluation metrics changed their scores significantly.
Which review-evaluation metrics were robust to rewriting?
Six: ReviewEval's Total Insights, Actionable Insights, and Actionability Score, and REMOR's Criticism, Materials & Methods, and Results & Discussion. These anchor in concrete, identifiable review content rather than holistic impressions, and passed an equivalence test (TOST) with bound δ=0.2σ.
Did AI rewriting make reviews look better or worse to the LLM judges?
Both, depending on the metric family. Dimensions from RottenReviews rose (e.g., Overall Score +5.573 on a 0–100 scale) and ReviewEval/REMOR mostly rose, while ScholarPeer — which scores a review against a paired human-written reference — scored the rewrites lower (e.g., Analytical Depth −1.192 on a 1–10 scale).
Is the code and dataset for this study publicly available?
Yes, per the paper: all prompts, results, and implementation code are in the public GitHub repository Reviewerly-Inc/llm_judge_reliability_analysis, and the review pairs come from PeerPrism, a published public benchmark (SIGIR 2026). Replication requires paid API access to GPT-5-mini and Gemini-2.5-Flash.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (log-scale scatter of Wilcoxon vs TOST p-values) is assessed only from its caption and in-text description; point-level details are not verifiable.
- Table 1's Gemini-2.5-Flash agreement column may have lost marks in text extraction: §1 claims 24/29 agreement (5 disagreements) but only 3 '—' entries are visible in the extracted table.
- Supplementary materials (annotation guidelines, repository contents) are referenced but not included in the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.