When Explanations Cannot Be Read: Measuring and Correcting SHAP and LIME Rendering for Right-to-Left Languages
Rameesha Zia, Rameesha Zia, Muhammad Shahid Iqbal Malik, Muhammad Shahid Iqbal Malik
cs.LG · cs.AI · cs.CL
Quick verdict
Sound, well-controlled: SHAP/LIME RTL labels unreadable (CER 0.82-0.98); OCR-measured raster fix reaches OCR ceiling; word-level only, human impact untested.
Summary
The paper addresses a readability failure rather than a modeling one: SHAP and LIME attribution values for right-to-left languages are mathematically valid, but their visualizations render labels out of reading order and with broken cursive letterforms. The authors characterize three failure mechanisms (directional inversion, shaping failure, layout asymmetry) across four languages (Urdu, Arabic, Persian, Hebrew) and eight plot families, then present SHAP-RTL, a rendering layer that performs complex text layout via HarfBuzz and FreeType with per-language font selection, rasterizing labels as images while preserving attribution values and feature ordering. Correctness is measured by a Tesseract OCR round trip over 200 feature words per language using character error rate (CER). Under Matplotlib 3.10.8, default rendering yields CER 0.820–0.979; the common reshape-and-reorder workaround fixes Arabic, Persian and Hebrew (CER 0.010–0.131) but fails for Urdu at 0.998 — worse than no correction — because Noto Nastaliq Urdu lacks Unicode presentation forms; SHAP-RTL reaches the OCR recognition ceiling in all four languages (Urdu 0.361 vs reference 0.374). Under Matplotlib 3.11.0, defaults improve but the workaround silently inverts (CER 0.815–0.981), while SHAP-RTL is version-invariant. Limitations: evaluation restricted to TF-IDF/logistic regression with word-level tokenization; layout asymmetry unresolved; OCR measures machine recoverability, not native-speaker comprehension; an LLM-based contextual explanation stage is provided but not evaluated.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Default SHAP/LIME visualizations render RTL feature labels that cannot be recovered from the figure (CER 0.820–0.979) while the attribution values remain mathematically valid — the defect lies entirely in the rendering path shared with Matplotlib.
- The widely used reshape-and-reorder workaround fails catastrophically for Urdu/Nastaliq (CER 0.998, worse than no correction) because Noto Nastaliq Urdu implements shaping via GSUB/GPOS tables and contains none of the Unicode presentation forms the workaround substitutes.
- Matplotlib 3.11.0's libraqm/HarfBuzz/SheenBidi rewrite fixes default rendering for Arabic and Persian but silently double-reorders pre-reordered workaround output in all four languages (CER 0.815–0.981), breaking adopted code on upgrade without any error.
- SHAP-RTL rasterizes labels via HarfBuzz/FreeType with per-language font selection, achieves CER at the OCR recognition ceiling in all four languages under both Matplotlib versions, and preserves attribution values, feature ordering and model outputs unchanged.
Strengths
Novel quantitative protocol for a previously anecdotal problem
The OCR round trip (Sec 6.4) — render each label as on a plot, read back with Tesseract 5.5.3, score CER over 200 stratified feature words per language — plus a same-pipeline OCR reference condition bounding per-language recognition accuracy, turns rendering failure into a measurable quantity (Tables 5–6) rather than inspection-based claims.
Well-controlled four-condition design
Conditions A (default), B (reshape+reorder), C (independent HarfBuzz/FreeType reference), D (SHAP-RTL) run on identical fonts, hardware and OCR configuration under both Matplotlib 3.10.8 and 3.11.0, with paired-bootstrap comparison appropriate for bounded, right-skewed per-word CER (Sec 6.4, 7.1).
Mechanism-level diagnosis with community impact
Honest scoping and controls
Preservation claim is architecturally plausible
Weaknesses
Single, dated model regime
A third of the paper's own failure taxonomy is unfixed
Metric measures machine, not human, readability — and the ceiling is noisy
RQ2 is posed but never answered
Shrinking residual value post-3.11.0
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a genuine gap: SHAP/LIME explanations for right-to-left languages produce plots whose labels native readers cannot read, even though the attribution values are valid. The authors frame this as a visualization-layer defect (Sec 2.3 correctly locates the directional assumption in the Matplotlib text path, not in SHAP or LIME), decompose it into three mechanisms — directional inversion, shaping failure, layout asymmetry (Sec 3.1, Table 1) — and contribute (i) a quantitative OCR-round-trip metric, (ii) a mechanism-level diagnosis of the community workaround's Nastaliq failure, (iii) a version-regression analysis of Matplotlib 3.11.0, and (iv) SHAP-RTL, a label-level rendering layer with per-language font selection. RQ1 is answered; RQ2 (LLM verbalization) is posed but, as the authors concede, not evaluated.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why do SHAP and LIME plots look wrong for Arabic or Urdu text?
SHAP and LIME compute valid attribution numbers for any language, but they hand label strings to a plotting library (Matplotlib) that historically drew text left-to-right without Arabic-script shaping. Letters then appear disconnected and in reverse order. The paper measures this with an OCR round trip: default rendering yields character error rates of 0.820–0.979 across Urdu, Arabic, Persian and Hebrew, meaning a reader cannot recover the feature word from the figure even though the underlying values are correct.
What is the reshape-and-reorder workaround and why does it fail for Urdu?
It is a common trick (arabic-reshaper plus python-bidi) that converts letters to Unicode presentation forms and reorders text for display in tools without bidirectional support. It works for Arabic, Persian and Hebrew, but the paper shows it fails for Urdu Nastaliq with a character error rate of 0.998 — worse than doing nothing — because fonts like Noto Nastaliq Urdu implement shaping through GSUB/GPOS tables and contain none of the substituted presentation-form code points, producing missing glyphs.
Does Matplotlib 3.11 fix right-to-left text in SHAP plots?
Largely yes for direct text: Matplotlib 3.11.0 rebuilt its text pipeline on libraqm, HarfBuzz and SheenBidi, and the paper's measurements show default rendering improves to near the OCR recognition ceiling (e.g., Arabic 0.011, Urdu 0.344 CER). However, it silently breaks the reshape+reorder workaround (CER rises to 0.815–0.981 across all four languages) because already-reordered text gets reordered a second time. Per-language font selection, such as a Nastaliq font for Urdu, is still not automatic.
How was rendering correctness measured in this paper?
Through an OCR round trip: each of 200 feature words per language is rendered as it would appear on an attribution plot, read back with Tesseract 5.5.3 in single-word mode using the matching language model, and compared to the source string via character error rate (CER), after normalizing Unicode and stripping invisible control characters. A same-pipeline render at larger scale provides an OCR reference bounding each language's recognition accuracy, and a paired bootstrap tests differences between conditions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–18 appear as captions only in the extracted text; the paired visual comparisons carrying Sections 7.2–7.3 could not be inspected.
- Interpretation near the top of the scale is bounded by the OCR engine itself: the paper's own recognition ceilings (Urdu 0.374, Hebrew 0.236) are high and noisier than the corrected conditions.
- The GitHub repositories and requirements file were not inspected; package completeness and mirror equivalence are assumed as stated.
- Extraction appears otherwise complete (all sections, tables and references present), so quantitative judgments rest on solid ground.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.