AtomCite: Verification and Correction of Supplied Page-Level Citations in Multi-Page Documents
Chen Qian, Yimeng Wang, Yu Chen, Lingfei Wu, Andreas Stathopoulos
cs.CL · cs.AI · cs.IR
Quick verdict
First benchmark+framework for verifying/correcting page-level citations; rigorous compute-matched evaluation backs core claims but scope is closed-world.
Summary
The paper introduces AtomCite, an agentic framework that verifies and corrects page-level citations already attached to answers over multi-page document images: it parses an answer into (claim, cited page) pairs, checks each claim in parallel against the cited page's OCR text and image, and applies a deterministic repair policy (keep, fix-page, remove). For evaluation it presents DocCite, described as the first benchmark for this task: 928 deterministically validated injected instances plus 2,468 candidate natural errors harvested from six frontier- and efficiency-tier generators, of which a two-annotator audit confirms 1,909 as genuine. Across Gemini, Claude, and GPT backbones, AtomCite reaches 93.0-93.7% binary verification accuracy on the injected set, significantly above every OCR-only condition — including compute-matched controls — every staged-prompting variant, and the strongest text-detector baseline (+4.1 to +4.9pp over CiteEval-Auto). Its repair policy lifts citation precision from 34.3% to 87-90% while retaining about 92% of correct claims. With frozen prompts and zero training it improves Qwen2.5-7B and Llama-3-8B on five public hallucination benchmarks. The audit further shows that noisy automatic labels can reverse verifier rankings. Main limitations: a closed-world setting, English business/legal documents, localization-based primary labels whose gold-set expansion self-audits at 60.5% precision, and two significant comparisons that weaken to p=0.11 under worst-case exclusion scoring.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No existing benchmark supports evaluating whether a system can verify and correct a page-level citation already attached to an answer over multi-page document images (Sec. 1-2).
- Image-aware verification significantly outperforms every OCR-only condition — including layout-aware and compute-matched controls — and every staged-prompting or text-detector baseline across three model families (Tables 1-3).
- The deterministic typed repair policy lifts citation precision from 34.3% to 87-90% on the injected mix while retaining about 92% of correct claims (Table 5).
- Noise in automatic natural-error labels biases measured verifier accuracy enough to reverse system rankings, motivating human-audited benchmark labels (Sec. 5.6, Appendix B).
- With frozen prompts and zero training, AtomCite improves two open 7-8B backbones on five public hallucination benchmarks in all ten cells (Table 6).
Strengths
Deterministic primary labels with a separate human audit layer
No LLM judge assigns test labels (Sec. 3); Layer-1 page-localization is validated by two-annotator census waves (608 + 306 disputed items) plus a 50-item spot-check of the undisputed region (0% noise, Wilson CI [0, 7.3]). The quantified finding that 72.9% of labels (C) disputed were noise vs 2.3% elsewhere (Sec. 5.6, App. B) is a methodological contribution in itself.
Alternative explanations are tested, not assumed away
The conditions ladder changes one ingredient at a time: the compute-matched OCR control (A3) and the matched sequential chain (B3) both trail (C) significantly in every family (Table 2, all 18 paired differences Holm-significant), and the entire A-ladder is re-run under a stronger OCR engine (MinerU 2.5, Table A3) with all nine gaps still significant.
Honest sensitivity analysis and worst-case rescoring
Correction results decompose cleanly against baselines
Unusual reproducibility infrastructure
Weaknesses
Closed-world scope limits deployment relevance
Primary ground truth is localization, not entailment
Thin audit capacity and unstated annotator provenance
Natural-error pool is generator-dependent and family-overlapped
Weaker, partly deferred correction evidence on natural errors
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AtomCite targets a task the authors argue is unmeasured: verifying and correcting page-level citations **already attached** to answers over multi-page document images, in a closed world where the document is supplied as page images plus OCR and retrieval quality is excluded (Sec. 1). The five claims are: (1) DocCite is the first benchmark for this task; (2) image-aware verification beats every OCR-only and staged-prompting condition across Gemini/Claude/GPT, including compute-matched controls (93.7/93.0/93.0% binary accuracy, Table 1); (3) the typed repair policy lifts citation precision from 34.3% to 87-90% at about 92% retention (Table 5); (4) automatic label noise can reverse verifier rankings; (5) zero-shot transfer improves two open 7-8B backbones on five public hallucination benchmarks. Claims are explicitly scoped in Sec. 6 (no cross-domain generality, no open retrieval, English business/legal pages), which is to the authors' credit.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does AtomCite do?
AtomCite is an agentic framework that takes an answer already equipped with page-level citations to a multi-page document, splits it into (claim, cited page) pairs, verifies each claim against the cited page's OCR text and page image, and repairs the citation layer with a fixed policy: keep verified claims, repoint citations to a nearby supporting page, delete unsupported claims, and flag hard text-image conflicts for human review instead of deleting them.
What is the DocCite benchmark?
DocCite is presented by the authors as the first benchmark for verifying and correcting supplied page-level citations in document images. It combines 928 injected citation errors validated by deterministic document-wide scans (DocCite-Syn) with naturally occurring errors harvested from six frontier- and efficiency-tier LLM generators (DocCite-Nat), where a two-annotator audit confirmed 1,909 of 2,468 candidates as genuine errors. It is built on MP-DocVQA and DUDE documents.
How accurate is AtomCite?
On the injected benchmark (DocCite-Syn), AtomCite reaches about 93% binary verification accuracy with Gemini (93.7%), Claude (93.0%), and GPT (93.0%) backbones, beating every OCR-only condition including compute-matched controls. Its repair policy raises citation precision from 34.3% to 87-90% while keeping about 92% of correct claims. On audited natural errors it detects 97.3-98.6% of validated errors at 5.4-12.3% false alarms.
Does AtomCite require training or fine-tuning?
No. AtomCite runs with frozen prompts and no training on top of frontier multimodal models. With the image channel and correction policy removed, the same verification design transferred zero-shot to five public text hallucination benchmarks, improving Qwen2.5-7B and Llama-3-8B over a same-backbone single-call judge in all ten cells tested.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The metadata abstract is cut mid-sentence ('AtomCite also t'), but the embedded full text is complete through references and appendices, so no content relied on in this review is missing.
- Figure 1 and the funnel diagram were extracted as interleaved plain text, so some figure-internal correspondences were read from captions rather than the graphics.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.