Who Pays for Open Review? Visible Author Reputation and Its Effect on Ratings
Qinghua Zhao, Xinyu Chen, Yanhui Yang, Tengfeng Sun, Junfeng Liu, Zhongfeng Kang
cs.DL · cs.AI
Quick verdict
Claims visible reputation steepens ICLR 2026 rating slopes, backed by an AI-reviewer simulation; large-scale but selection-confounded, effects tiny.
Summary
Motivated by a November 2025 OpenReview bug that broke reviewer anonymity, this paper measures how visible author reputation affects review scores. Using the full ICLR 2026 record (18,789 rated and 13,726 decided submissions), papers are split into de facto open and blind groups by whether an arXiv preprint appeared before review start, and authors are linked to reputation proxies: a 10-tier institution mapping from QS rankings plus LLM-graded company tiers, and four OpenAlex bibliometrics via ORCID. Ratings rise with reputation in both groups, with a steeper open-review slope for the institution proxy (γ=0.021, p=0.008 mean-author; p=0.041 max-author), concentrated near the acceptance borderline (thresholds ~3.5 blind, ~4.5 open). A controlled simulation rates 600 stratified submissions under fixed manuscripts with low/middle/high stated reputation using five frontier models: four of five show 0.2–0.6 point rises (0.5 for claude-opus-5), robust across prompts and input scopes, while gpt-5.6-sol shows no effect. Key limitations, acknowledged by the authors: arXiv posting is self-selected and open-group authors are more reputable (Fig. 7), so observational estimates are upper bounds; bibliometric proxies are untested for slope differences due to sparse ORCID coverage; bibliometrics are May-2026 snapshots; no code or data release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Ratings rise with author reputation under both de facto blind and de facto open review at ICLR 2026, and the reputation slope is significantly steeper under open review for the institution proxy (mean-author p=0.008; max-author p=0.041; first-author p=0.306).
- The reputation effect concentrates at borderline ratings: the change in P(rating ≥ c) peaks around thresholds 3.5 (blind) and 4.5 (open) and nearly vanishes at scale extremes.
- Holding the manuscript fixed, five frontier AI reviewers rate the same paper higher under a high-reputation stated author — 0.2–0.6 points in four of five models (0.5 for claude-opus-5) — with gpt-5.6-sol the exception.
- Because arXiv preposters are self-selected and more reputable (Fig. 7), the observational open−blind gap is an upper bound on the reputation effect, not a causal estimate.
Strengths
Unmatched observational scale for this question
Full ICLR 2026 crawl via OpenReview API v2 (§4.1): 19,814 submissions → 18,789 rated, 13,726 decided (39.0% accepted, Table 1), an order of magnitude beyond the few-hundred-paper randomized trials the related-work section contrasts against (§1, §2).
Falsifiable analytical framework, confirmed where it bites
Eq. (1)–(2) predict a transfer (gain above pool mean π̄, loss below) and that effects concentrate where rating density is highest; §5.2's threshold curves show exactly this inverted-U (open coefficient 0.002 at threshold 2.0 → 0.025 at 4.5 → 0.003 at 7.0 for institution/max-author), peaking at ~3.5 blind vs ~4.5 open.
Observational and experimental evidence triangulate
Unusually thorough robustness grid
Candor about the key confound
Weaknesses
Headline identification is confounded by self-selection
Effect size is small relative to framing
'De facto open' is an intent-to-treat label
Simulation lacks dispersion reporting and shows internal inconsistency
No multiple-comparison control and no code/data release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks what happens to review scores when author identity becomes visible, motivated by the November 2025 OpenReview bug that exposed author–reviewer linkages across ICLR, ACL, KDD and ICML. On the complete ICLR 2026 record (§4.1: 19,814 crawled; 908 desk-rejected; 18,789 rated; 13,726 decided, 39.0% accepted per Table 1), submissions are split into "de facto open" and "de facto blind" by whether the first arXiv version predates the 2025-10-10 review start (Table 2, decided subset: 5,189 open / 1,837 ambiguous / 6,700 blind). Three claims carry the paper: (1) ratings rise with reputation under both mechanisms but steeper under open — significant only for the institution proxy (Table 4: mean-author p=0.008, max-author p=0.041, first-author p=0.306); (2) the effect sits at the acceptance borderline (Fig. 2: inverted-U peaking at ~3.5 blind, ~4.5 open); (3) an AI-reviewer simulation with the manuscript fixed reproduces a 0.2–0.6 point rise in four of five models (claude-opus-5: 0.5; gpt-5.6-sol: none).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this ICLR 2026 peer review study?
That review scores rise with author reputation whether or not author identity is effectively hidden, but the rise is steeper when a public arXiv preprint makes authors identifiable — significantly so only for the institution-based reputation proxy (p=0.008 under mean-author aggregation) — and the extra effect concentrates near the accept/reject borderline (thresholds ~3.5–4.5).
How large is the reputation bias reported in the paper?
Small observationally: 0.06–0.19 rating points across the whole observed reputation range on ICLR's 0–10 scale (slope differences γ=0.007–0.021). In the AI-reviewer simulation, moving the stated author from low to high reputation raised scores by 0.2–0.6 points in four of five models (0.5 for claude-opus-5); gpt-5.6-sol showed no effect.
Is the evidence in this paper causal?
No. The authors show that authors who post arXiv preprints are already more reputable (e.g., QS mean rank 343 vs 427 under mean-author aggregation), so the open−blind comparison is confounded by self-selection and is presented as an upper bound; only the fixed-manuscript AI simulation isolates reputation, and it uses coarse stated descriptors rather than real identities.
Are the data and code from this study available?
The paper describes fully public sources (OpenReview API v2, a Kaggle arXiv metadata snapshot downloaded 2026-05-14, and OpenAlex) in enough detail to rebuild the corpus, but the text contains no statement releasing code or data.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are present only as extracted axis labels and captions; numeric 95% bands and per-bin means (Fig. 1, 2, 4–7) cannot be read from the provided text.
- The exact regression specification (covariates, standard errors, clustering) is not stated anywhere in the text; Table 4 reports only slopes and p-values.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.