HalluPeer: A Taxonomy-driven Benchmark for Detecting Hallucinations in Scientific Peer Reviews
Tzu-Ling Lin, Dong-Ting Yao, Teng-Fang Hsiao, Wei-Chih Chen, Hong-Han Shuai
cs.AI · cs.CL
Quick verdict
Claims first peer-review hallucination benchmark; rigorous taxonomy pipeline, but fine-tuned detector gains may partly reflect synthetic-injection artifacts.
Summary
HalluPeer is a benchmark for detecting hallucinations in scientific peer reviews, built as aligned triples of paper content, human-written reviews, and hallucination-injected reviews annotated for detection, type classification, and span localization. The authors induce a hierarchical taxonomy — 265 nodes, 205 leaves, nine coarse anchors such as Number and Attribution Failure — through LLM-guided recursive decomposition, multi-model consensus filtering, and expert validation. From 12,000 OpenReview ICLR/NeurIPS papers with 38,063 reviews, an automated pipeline injects taxonomy-directed hallucinations into review sentences, filtered by compatibility and semantic-equivalence checks. Across the three tasks, zero-shot LLMs including GPT-5.2 struggle: review-level type classification tops out at 0.19 Macro-F1, and detection MCC at 0.25, while QLoRA fine-tuned Qwen3-32B reaches 0.90/0.91 F1 (review/sentence), 0.59/0.82 classification Macro-F1, and 0.91 Token-F1 localization. Cross-venue and cross-generator ablations show stability. On 1,161 manually annotated authentic NeurIPS 2024 reviews containing 20 genuine hallucinations, the fine-tuned detector achieves TPR 100% at FPR 22.1%, versus TPR 85% at FPR 0.3% for zero-shot Claude Opus 4.7. Main limitations: synthetic injections may carry learnable artifacts, meta-review-based filtering may bias base reviews, and the transfer claim rests on only 20 positive examples.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Peer-review hallucination detection requires paper-grounded verification with an annotation structure no prior resource provides.
- Zero-shot LLMs and verifiers fail to separate hallucinated review claims from legitimate critique: review-level type classification peaks at 0.19 Macro-F1 and detection MCC at 0.25.
- QLoRA fine-tuning on HalluPeer lifts performance dramatically (Qwen3-32B: 0.90/0.91 detection F1, 0.91 Token-F1 localization) with stable cross-venue and cross-generator transfer.
- HalluPeer-defined patterns occur in real reviews; a synthetic-only trained detector recovers all 20 expert-annotated authentic hallucinations at FPR 22.1%.
Strengths
Clear problem formalization
Sec. 1 and 5.1 define review hallucination strictly as a paper-grounded claim failure, explicitly excluding tone and subjective judgment; this exclusion is embedded consistently in every evaluation prompt (Figs. 17-21).
Taxonomy construction with multi-stage quality control
Sec. 3 + Appendix F: three proposer LLMs generate trees independently; consensus filtering reduced 86,800 candidate node pairs to 434 high-overlap pairs for expert adjudication, yielding a 265-node/205-leaf tree (Table 6) whose operational leaves serve both as injection instructions and evaluation labels.
Robustness ablations rare for benchmarks
Grounding in authentic reviewer errors
Appropriate metrics under imbalance
Weaknesses
Generator circularity only partially addressed
Thin authentic positive base
Selection bias from meta-review alignment filter
Review-level classification stays weak after fine-tuning
No statistical significance testing shown
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
HalluPeer — Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the HalluPeer benchmark?
HalluPeer is a research dataset for training and evaluating systems that detect hallucinations — factually unsupported or incorrect claims — in scientific peer reviews. It aligns each paper with its human-written OpenReview reviews and LLM-injected hallucinated versions of those reviews, labeled for detection, hallucination-type classification (a 265-node taxonomy), and span localization, built from 12,000 ICLR and NeurIPS papers and 38,063 reviews.
How common are hallucinations in real peer reviews?
In the paper's manual annotation of 1,161 real NeurIPS 2024 reviews, expert annotators found 20 genuine reviewer hallucinations — roughly 1.7% prevalence. A separate case study over 13,803 NeurIPS 2024 reviews validated 11 hallucination instances among 200 detector-flagged candidates.
Can current LLMs detect hallucinated peer-review claims out of the box?
According to the paper, no: zero-shot LLMs up to GPT-5.2 reach at best 0.19 review-level Macro-F1 on hallucination-type classification and about 0.25 MCC on detection, often confusing unsupported claims with legitimate critique. Fine-tuned models reach 0.90+ F1, but they were trained on the same synthetic injection distribution.
Where can I get HalluPeer's code and data?
The paper points to a project page at github.com/Lin-TzuLing/HalluPeer, and the appendices publish construction prompts, model checkpoints, and training hyperparameters. Whether the full dataset is downloadable cannot be verified from the provided text excerpt alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The excerpt is a sampled 80k-char extraction with explicit sampling gaps; Sec. 5.2-5.4 settings text, the specialized-verifier result rows (HHEM/True-NLI/RefChecker), Appendix E per-label tables and Table 8 contents were skipped, so judgments about specialized verifiers rely on the authors' prose rather than their numbers.
- Sec. 8 Limitations cuts off mid-item at the word 'Synthetic'; the remaining limitation items sit inside a sampling gap.
- Table 4's column headers are not visible in the extraction; only papers (12,000) and reviews (38,063) from it are cited here, matching the abstract.
- Figures appear as fragmented extracted text; layout-dependent details were inferred from captions only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.