When Superpixels Fail on Documents: A Study of Segmentation for LIME Explanations
Quentin Telnoff, Emanuela Boros, Mickaël Coustaty, Robin Jarry, Fabrice Crohas, Antoine Doucet
cs.CV · cs.AI
Quick verdict
Paper shows LIME segmentation drives explanation quality on documents; RVL-CDIP study is convincing but narrow: 3 classes, one ResNet-50, 60-image bias test.
Summary
The paper examines how the segmentation step of LIME — which defines the interpretable regions being perturbed — affects explanation quality for document image classification. Using RVL-CDIP with a ResNet-50 classifier, the authors compare standard superpixel methods (Quickshift, SLIC) against document-aware segmentations built from OCR bounding boxes (PaddleOCR) combined with regular grids, across three classes (Invoice, Budget, Form). Three metrics are used: consistency (Spearman correlation across repeated runs), correctness (insertion/deletion AUC), and local fidelity (surrogate R²). Document-aware segmentations dominate: BBs only reaches 0.933 Spearman consistency, 0.764 insertion AUC, and 0.031 deletion AUC, versus 0.432 consistency and 0.080 insertion for Quickshift. Consistency converges with far fewer perturbations for low-dimensional, structure-aligned segmentations. The authors further show that BBs-only explanations expose the known RVL-CDIP identification-code shortcut (top-ranked region in 15/60 images) while SLIC pushes the code to an average rank of ~39, obscuring it. Limitations include evaluation restricted to three of sixteen classes, a single backbone, no significance testing, and a 60-image bias analysis; the classifier's test accuracy is also unreported.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The segmentation function S is not neutral preprocessing: it defines the interpretable space of LIME and is a main driver of explanation consistency, correctness, and local fidelity on document images.
- Document-aware segmentations (OCR bounding boxes, coarse grids) outperform Quickshift and SLIC on all three metric families for RVL-CDIP Invoice/Budget/Form classification.
- Lower-dimensional, structure-aligned segmentations converge to stable explanations with far fewer perturbation samples, framing LIME as a local estimation problem whose sample complexity is set by segmentation.
- BBs-only explanations expose the known RVL-CDIP identification-code shortcut (top-ranked region in 15/60 images) while SLIC obscures it, degrading the code to an average rank of ~39.
Strengths
Well-posed question with an explicit formalization of the mechanism
Section 3.2 derives precisely why the segmentation function S changes the explanation problem: it sets the semantic granularity of interpretable units, the dimensionality d of the surrogate regression, and the plausibility of perturbations — turning an intuition into a testable frame.
Systematic nine-variant comparison across three complementary metric families
Table 1 spans consistency (Spearman), correctness (insertion/deletion AUC), and local fidelity (surrogate R²) for Quickshift, SLIC, BBs only, Grid 4×4 w/o BBs, and five BBs-with-grid variants; consistency falls monotonically with interpretable dimensionality (0.975 → 0.615 from 4×4 grid to 30×10 grid), exactly as the sample-complexity framing predicts.
Honest limitation reporting
Bias-discovery claim supported by both qualitative and quantitative evidence
Weaknesses
Narrow evaluation scope relative to the general claims
No inferential statistics; key sample sizes unstated
Classifier quality is unreported
Unaddressed counter-evidence in the correctness results
OCR dependency unexamined
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed question: does the segmentation step of image-based LIME — the choice of interpretable units being perturbed — materially change explanation quality on document images? Section 3.2 formalizes why the segmentation function S determines the dimensionality d, semantic granularity, and perturbation plausibility of the explanation problem, and Section 3.3 compares two superpixel baselines (Quickshift, SLIC) with document-aware alternatives built from PaddleOCR bounding boxes, optionally combined with regular grids. Three central claims are made: (1) segmentation is a main driver of LIME explanation quality on documents; (2) document-aware segmentations improve consistency, correctness, and local fidelity; (3) they expose the known RVL-CDIP identification-code shortcut that superpixel LIME obscures. Claims (1) and (2) are appropriately scoped to the evidence; claim (3) rests on a 60-image analysis.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why does LIME produce poor explanations on document images?
Standard image LIME uses superpixel algorithms (SLIC, Quickshift) designed for natural images; on documents they fragment text and layout into arbitrary visual fragments, so the perturbed units do not correspond to meaningful document structure. This paper reports Spearman consistency of only 0.432 for Quickshift versus 0.933 for OCR bounding-box segmentation on RVL-CDIP, along with much lower insertion AUC (0.080 vs. 0.764) for the superpixel method.
What segmentation should I use for LIME on document images?
Based on this RVL-CDIP study, OCR bounding boxes alone ('BBs only', produced with PaddleOCR) were best overall: highest insertion AUC (0.764), lowest deletion AUC (0.031), high consistency (0.933), and the ability to reveal the dataset's ID-code shortcut. A coarse 4×4 grid without OCR boxes was the most consistent (0.975) but slightly less faithful, while fine grids (10×10 and denser) degraded to or below SLIC's stability.
What is the RVL-CDIP identification-code bias?
Many RVL-CDIP documents contain class-specific identification numbers that deep models exploit as shortcuts instead of learning document structure; prior work (Saifullah et al. 2024; Larson et al. 2025) estimated this affects roughly 25–50% of predictions and that standard LIME fails to reveal it. This paper shows OCR-box LIME ranks the code as the top region in 15 of 60 examined images, while SLIC pushes it to an average rank of ~39.
What are the main limitations of this segmentation study?
Explanation experiments cover only 3 of RVL-CDIP's 16 classes with a single ResNet-50 classifier on one dataset; no statistical significance tests or confidence intervals are reported; the bias-discovery analysis uses just 60 images; the classifier's test accuracy is never reported; and no code release is mentioned.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–6 are present only as captions in the extraction; heatmap/overlay content (e.g., the Figure 6 explanation images) and the exact curve values in Figures 4–5 could not be assessed directly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.