ReGround: Grounding Reviewer Comments in Multimodal Evidence
Serwar Basch, Lizhen Qu, Iryna Gurevych
cs.CL · cs.IR
Quick verdict
Rebuttal-derived dataset grounds reviewer comments in multimodal paper evidence; solid QA and analysis, but author-cited-only gold caps score meaning.
Summary
ReGround is a large-scale dataset linking 10,267 reviewer comments to 16,274 evidence units (paragraphs, sections, figure/table captions and images) in 3,656 original anonymous submissions from NLPeer v2 (EMNLP, COLING, NAACL, ACL 24–25). The authors exploit a simple observation: author rebuttals explicitly cite the submission content used to answer reviewers, providing high-precision grounding labels. A pipeline of regex reference detection, LLM-based (gpt-oss-120b) rebuttal-to-review alignment, reference resolution against parsed PDFs, and precision-oriented filtering yields the links; quality checks report 96% alignment acceptance (κ=0.83) and 97%/93% filter precision/recall. Casting grounding as retrieval, the paper benchmarks sparse, dense, cross-encoder, LLM pointwise, and multimodal rankers under four settings. Unified retrieval is hard (best MRR 10.87, Recall@10 21.15 with Qwen-3 30B Instruct); oracle evidence-type information helps substantially, but predicted-type hard filtering collapses R@10 to 6.11; multi-evidence retrieval stays weak (12.69 R@1); and for figures/tables, image and caption signals are complementary, with image retrieval significantly better for tables (McNemar p=0.0156). Key limitations: gold sets are author-cited evidence only, so relevant but uncited content counts as errors and absolute scores partly measure annotation coverage; only 15.24% of weakness/suggestion spans are covered; several validations use single annotators, including authors; and only zero-shot retrievers are evaluated.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Explicit paper references in author rebuttals provide a high-precision, automatically scalable annotation source for grounding reviewer comments (96% alignment acceptance, 97% filter precision).
- Retrieval over a heterogeneous evidence pool is hard for all model families, with the best LLM ranker (Qwen-3 30B Instruct) reaching only 21.15 Recall@10.
- Evidence-type inference is a dominant bottleneck: oracle type information helps substantially, but hard filtering by predicted type collapses R@10 from 21.15 to 6.11.
- Multi-evidence grounding remains difficult even under oracle type conditions (12.69 R@1 joint retrieval for the best model).
- For figures and tables, visual and caption-text signals are complementary, with image retrieval significantly better for tables (McNemar p=0.0156).
Strengths
First large-scale resource for a genuinely new task setting
Table 1 shows ReGround is the only dataset combining original submissions, multimodal evidence, automatic annotation, and 3,656-paper scale; prior resources (ARIES 42 papers, CLAIMCHECK 41, F1000RD 172) are small, manual, or text-only, and target revisions rather than the reviewed submission.
Unusually rigorous auditing of an automatic pipeline
Section 3.6 reports 96% alignment acceptance (κ=0.83, 100 papers), 97% precision/93% recall for the LLM filter (κ=0.84, 300 pairs), and manual inspection of 200 comments/316 references finding only 1.58% coarse spans; Appendix K quantifies coverage (15.24% of weakness/suggestion spans) and selection bias (Table 10) instead of hiding it.
Evaluation design cleanly factors the difficulty
Concrete error analysis with statistics where it matters
Weaknesses
Gold-standard completeness is assumed, not measured
Coverage and selection bias limit external validity
Single-annotator validations, some circular
Zero-shot only, strict scoring
GPT-5.1 compared from a 50% subset
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ReGround (arXiv 2609.11460) introduces a 3,656-paper dataset connecting 10,267 reviewer comments to 16,274 evidence units (13,391 unique) — paragraphs, sections, figure/table captions, and figure/table images — in original anonymous submissions from NLPeer v2 (EMNLP 24/25, COLING 25, NAACL 25, ACL 25). Grounding is cast as retrieval from a per-paper candidate pool, targeting the implicit setting where comments contain no explicit references. Central claims: (1) rebuttal-cited references are a high-precision automatic annotation source; (2) unified retrieval is hard (best 21.15 R@10, Table 3); (3) evidence-type inference is a dominant bottleneck; (4) multi-evidence retrieval stays weak under oracle type; (5) image and caption signals are complementary. The claims are mostly supported by the presented experiments, with one structural caveat about what the gold standard actually measures.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ReGround dataset?
ReGround is a benchmark dataset linking 10,267 peer-reviewer comments to 16,274 pieces of evidence (paragraphs, sections, figure/table captions, and figure/table images) in 3,656 original anonymous paper submissions, built from the public NLPeer v2 corpus covering EMNLP, COLING, NAACL, and ACL venues. The task is retrieval: given a reviewer comment, find the paper content that addresses it.
How were the ground-truth links in ReGround created without manual annotation of everything?
The authors noticed that author rebuttals often explicitly cite parts of the submitted paper (e.g., 'Section 3.2', 'Table 2') when answering reviewers. Regular expressions detect these citations, an LLM (gpt-oss-120b) aligns each rebuttal sentence to the reviewer comment it answers, and the citations are resolved against the parsed PDF. Filters remove quoted review text, trivially explicit cases, and content not actually in the submission; manual checks report 96% alignment acceptance and 97% filter precision.
Why are the retrieval benchmark scores in the ReGround paper so low?
Three reasons combine: reviewer comments are implicit and evaluative rather than keyword-like queries (average comment-evidence cosine similarity is only 0.377); models must also infer whether the answer is a paragraph, section, caption, figure, or table — hard type filtering by predicted type drops the best model's R@10 from 21.15 to 6.11; and scoring requires exact unit matches with no partial credit. Additionally, the gold set is only the evidence authors cited, so any relevant but uncited content counts as an error, pushing absolute numbers down.
Is the ReGround dataset and code publicly available?
Yes. The paper states code and data are publicly released at github.com/UKPLab/emnlp2026-reground under CC-BY-NC 4.0, consistent with the NLPeer source data license. The dataset is restricted to NLP papers because peer-review data from other domains was not accessible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- In-figure text of Figures 2–3 is extracted out of reading order (axis labels and counts interleaved); all numbered tables, body text, and both prompt figures are intact — no truncation observed at either end of the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.