How Much Were You Told? Measuring External Information in Peer Reviews
Matthieu Dubois, Pablo Piantanida, François Yvon
cs.CL
Quick verdict
Self-Conditioning: unsupervised external-information score for peer reviews; separates delegated from polished reviews, but validated on synthetic regimes only.
Summary
The paper reframes the assessment of LLM-assisted peer review: rather than detecting machine-shaped surface text, it estimates how much information a review carries beyond its production context — the manuscript plus a generic reviewing instruction. The proposed Self-Conditioning estimator compares a review's log-likelihood under this baseline context with its likelihood when the context is augmented with hints sampled from the review itself (default: 30% of words, 10 draws), scored by Llama-3.1-8B-Instruct. On IntelLabs-500 (500 ICLR 2019 human reviews with counterparts from five generators, plus new GPT-5.4 and Gemini-3.1-Pro generations), it separates fully-delegated from human reviews with AUC 0.937-1.000, while machine-polished reviews stay near the human distribution (AUC 0.556-0.610). Along a ladder of reviews regenerated from 1-8 human-derived bullet points, AUC against human decreases monotonically (r=-0.99) while EditLens remains saturated at 1.00. Acknowledged weaknesses include sensitivity to the scoring model and disclosure budget for the newest generators, very low true-positive rates at strict false-positive thresholds (GPT-5.4: 0.6% at 0.1% FPR), and evasion via temperature 2.0 at the cost of coherence. Main limitations: all regimes are synthetic, human baselines are pre-2022 ICLR reviews, provenance remains unidentifiable, and no confidence intervals accompany the AUC estimates.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Peer-review policy compliance depends on who or what supplied the review's content, an axis that existing ATD methods, which measure surface form, do not capture.
- Self-Conditioning, an unsupervised likelihood-gain estimator with hints drawn from the review itself, separates fully-delegated from machine-polished reviews with AUC up to 1.0 (IntelLabs-500, Table 1).
- The score decreases monotonically as the generator is given more external information (bullet ladder, r=-0.99, Fig. 2/Table 3), unlike EditLens which stays saturated at 1.00.
- High-temperature sampling (T=2.0) can push delegated reviews into the human score range, but at severe cost to coherence, so evasion is detectable by inspection (§6, App. I).
Strengths
Problem formulation matches the actual policy question
§1/§2 distinguish content delegation (prohibited) from surface polishing (permitted), a boundary ATD cannot enforce; the external-information target and its source-agnostic interpretation are stated precisely, with the Markov decomposition of Eq. 4 conceding up front that stochastic generation can inflate the score.
Strong unsupervised separability including frontier generators
Table 1/Fig. 1: AUC vs. H of 0.954 (GPT-5.4) and 0.937 (Gemini-3.1-Pro) where TMR-AI, ModernBERT, Binoculars and Fast-DetectGPT fall to 0.507-0.825; vs. MP the method holds 0.922-0.941 while Binoculars/Fast-DetectGPT confuse classes.
Graded validation with a proper control
Unusually honest failure-mode and ablation reporting
Careful theory-practice separation
Weaknesses
Entirely synthetic evidence base
No statistical uncertainty anywhere
Human anchor is temporally stale and potentially contaminated
Polishing-insensitivity shown with small rewriters only
Screening operating point is weak for the newest generators
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a gap between peer-review policy and detection technology: venues forbid *delegating* the critique to an LLM while permitting surface polishing, but Artificial Text Detection (ATD) measures surface provenance and cannot enforce that boundary. The authors reformulate assessment as estimating the **external information** of a review — information not explained by the production context C (paper + generic reviewing instruction, Appendix J) — and propose **Self-Conditioning** (Eqs. 3, 5-6): the per-token log-likelihood gain of a review when C is augmented with hints sampled from the review itself (default: 30% of words in original order, N=10 draws), scored with Llama-3.1-8B-Instruct. Central claims: (1) the score separates fully-delegated (FD) from machine-polished (MP) reviews with AUC up to 1.0; (2) it varies monotonically with externally supplied information; (3) standard ATD measures a different axis. The information-theoretic framing is careful: Appendix A gives coding, likelihood-ratio, and population (conditional mutual information) interpretations, and Eq. 4's decomposition I(T;X|C) = I(U;X|C) + I(T;X|C,U) concedes that stochastic generation can inflate the score.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'Self-Conditioning' in this paper?
It is an unsupervised score that measures how much extra information a peer review contains beyond the reviewed paper and a generic reviewing instruction. The method compares the review's likelihood under the paper-only context with its likelihood when hints sampled from the review itself (by default 30% of its words) are added to the context. Reviews whose content the context cannot explain gain much more from their own hints, so they score higher.
Can this method prove a review was written by a human or by an LLM?
No. It measures information not explained by the declared context, regardless of where that information came from. A high score does not establish human authorship and a low score does not prove LLM delegation. The authors position it as a screening aid that flags reviews for human inspection, not as a standalone misconduct detector.
How well does it perform?
On 500 ICLR 2019 reviews and their LLM-generated counterparts, it separates fully-delegated reviews from human ones with AUC between 0.937 and 1.000 across seven generators, including recent frontier models. Reviews that humans wrote and an LLM only polished score close to human reviews (AUC 0.556-0.610), and the score falls monotonically as the generator is given more human-supplied content — behaviour standard AI-text detectors do not show.
Can the method be fooled?
Yes. Generating a delegated review at sampling temperature 2.0 pushes its score into the human range. However, the resulting text shows severe degradation in coherence, grammar, and logical consistency, making such reviews conspicuous to a human reader, so the evasion trades detectability by the score for visibility by inspection.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are unavailable as images; distribution-shape judgments rely on reported AUC values and figure captions rather than visual overlap assessment.
- Equations were extracted from PDF text and may contain minor OCR artifacts (e.g., Eq. 5 summation formatting); all cited numbers were cross-checked against the tables.
- Appendix H (other years/NeurIPS) reports results only as figure AUC annotations; per-year sample sizes are not stated in the extracted text.
- The IntelLabs dataset carries an Intel internal-research-use license (Table 18), which may constrain third-party reproduction of the exact data split even though the scoring code is released.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.