LeakageBench: Document-Level Leakage Risk for Redacting Personally Identifiable Information in Document Images
Vishnu Prasad Vijaya Kumar, Santhosh Venkatesh, Ivan P. Yamshchikov
cs.CV · cs.CL
Quick verdict
A 500-page document-image PII redaction benchmark finding near-total page leakage (96.8%); thorough diagnostics, but author-annotated with no uncertainty.
Summary
The paper introduces LeakageBench, a challenge set of 500 business-document images with 11,954 GDPR/CPRA-aligned PII annotations grouped into direct identifiers, linkage keys, and contextual re-identification surfaces. Under a unified image-space bounding-box protocol (IoU 0.75), the authors propose release-oriented document-level leakage metrics (DocLeak) reported alongside entity-level localization and typing F1. They evaluate nine OCR-dependent pipelines (Presidio, GLiNER variants, Google DLP, Amazon Comprehend, OpenAI Privacy Filter over Tesseract/Textract) and five OCR-free vision-language configurations (Qwen3-VL-32B, InternVL3-38B, GPT-5.4, GPT-5.5, GPT-5.5 with Code Interpreter). Results are sobering: the best localization F1 is 0.304 (Amazon Comprehend PII), the best typed F1 is 0.119 (GPT-5.5 with Code Interpreter), and even the strongest configurations leak critical Direct+Linkage identifiers on 96.8% of applicable pages, with leaking pages retaining 5.3–8.7 critical identifiers on average. Diagnostics attribute OCR-dependent failures mainly to box/alignment errors (45.9–84.5% of attributable leaks) and VLM failures to sub-threshold spatial grounding (up to 98.9%). Annotation quality is supported by Cohen's κ of 0.979 on a 10% cross-annotated subset. Limitations include author-only annotation, no confidence intervals or significance testing, limited ecological validity from public-source documents, a DUA-gated release, and the absence of trained layout-model baselines.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing PII benchmarks are text-centric and do not measure page-level release safety; LeakageBench fills this gap with 500 document images and 11,954 GDPR-aligned annotations.
- All 14 evaluated configurations, including commercial detectors and frontier VLMs, leak critical Direct+Linkage identifiers on at least 96.8% of applicable pages.
- Code Interpreter raises GPT-5.5 localization F1 from 0.090 to 0.249 and typed F1 from 0.050 to 0.119, yet reduces DocLeakcrit only from 0.990 to 0.968.
- OCR-dependent failures are dominated by box/alignment errors (45.9–84.5% of attributable leaks for 7 of 9 systems) rather than detector misses, and Textract alone cannot recover 14.6% of value-bearing critical annotations.
- Schema coverage alone does not solve the task: full-schema systems still leak on more than 96% of Linkage pages (Table 4).
Strengths
Release-safety framing fills a real evaluation gap
DocLeak (Sec. 3.4) measures the fraction of pages with any unmatched in-scope identifier, complementing span-level F1 that text-centric benchmarks (TAB, SPEDAC, i2b2) and extraction benchmarks (FUNSD, SROIE, BuDDIE) omit; the asymmetric FP/FN trade-off motivating this is well argued in Sec. 1.
Diagnostic depth is unusually thorough
Schema coverage is separated from detection (Tables 4, 8), OCR recoverability is measured detector-independently (Textract misses 623/4,280 = 14.6% of value-bearing critical values vs Tesseract 17.9%, Table 15), and leaked identifiers are exclusively attributed to OCR-miss/detector-miss/box-alignment buckets (Tables 5, 14: box/alignment is 45.9–84.5% for 7 of 9 OCR-dependent systems).
Headline conclusion is robustness-checked
Above-average disclosure for reproduction
Weaknesses
Author-only annotation with thin QC sample
No statistical uncertainty anywhere
Baselines may be under-tuned relative to the harsh protocol
Ecological validity of the privacy threat model
Trained document-AI baselines excluded
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
LeakageBench (arXiv 2609.02207) targets a gap the authors argue existing evaluation leaves open: text-centric PII benchmarks (TAB, SPEDAC, i2b2) and document-understanding benchmarks (FUNSD, SROIE, BuDDIE) do not measure whether a document *page* is safe to release after redaction. The contribution is a 500-page challenge set of business-document images (OCR-IDL/UCSF 113 pages, VRDU Ad-Buy 222, FCC free-form 165) with 11,954 boxed PII annotations — 4,315 critical Direct+Linkage identifiers, 8.63 per page (Table 2) — a GDPR/CPRA-anchored three-group schema (Table 1), and a release-oriented metric, DocLeak: the fraction of pages with at least one unmatched in-scope identifier, reported alongside page-macro F1loc and F1type under a unified image-space box interface at IoU τ=0.75.
Central claims: (1) all 14 evaluated configurations, from Presidio to GPT-5.5 with Code Interpreter, leak critical identifiers on ≥96.8% of applicable pages (Table 3); (2) tool assistance improves localization (GPT-5.5 F1loc 0.090→0.249, F1type 0.050→0.119) without achieving page safety (DocLeakcrit 0.990→0.968); (3) OCR-dependent failures are dominated by box/alignment errors (45.9–84.5% of attributable leaks for 7 of 9 systems, Table 14), while OCR-free failures are dominated by sub-threshold boxes (up to 98.9% for GPT-5.5+CI, Table 5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LeakageBench?
It is a benchmark of 500 document images (business documents such as invoices, emails, forms, and letters) containing 11,954 human-annotated PII regions. Unlike text-only PII datasets, it measures whether a redaction system makes an entire page safe to release — a page counts as leaked if even one critical identifier remains undetected.
What are the main results of the paper?
All 14 evaluated systems leak critical identifiers (names, IDs, contact and linkage keys) on at least 96.8% of applicable pages. The best localization F1 is 0.304 (Amazon Comprehend over Textract OCR), and GPT-5.5 with Code Interpreter achieves the best typed F1 of 0.119 while still leaking on 96.8% of pages. Better detection improves average scores but does not make pages safe.
Why do PII redaction systems fail on document images?
For OCR-based pipelines, the dominant failure is placing or reconstructing bounding boxes wrongly even when the text detector fires (45.9–84.5% of attributable leaks for 7 of 9 systems). For vision-language models, predicted boxes overlap the target but fall below the IoU 0.75 threshold (up to 98.9% of residual leaks for GPT-5.5 with Code Interpreter). OCR itself also misses 14.6–17.9% of value-bearing identifiers outright.
Is the LeakageBench dataset publicly available?
The paper states that the document images and annotations will be made available for research use under a privacy- and license-aware Data Use Agreement (DUA). No code repository is mentioned in the text, so the evaluation harness availability is currently unclear.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels (Figures 1–3) appear only as captions in the extracted text; their visual content could not be reviewed.
- Appendices D–I are presented only as table captions and numeric tables in the extraction; some per-table prose commentary may be condensed relative to the PDF layout.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.