Evaluating In-Context Learning and Retrieval Strategies for Devanagari Post-OCR Correction
Abhishek Bhandari, Gaurav Harit
cs.CL · cs.CV
Quick verdict
First systematic ICL evaluation for Devanagari post-OCR correction; proposed CharBM25 beats random and matches dense retrieval, though synthetic-only evaluation and missing significance tests temper conclusions.
Summary
The paper presents the first systematic evaluation of in-context learning for post-OCR correction of Devanagari (Hindi, Marathi). The authors build a 20,000-sentence synthetic benchmark (Tesseract 5 render-and-OCR over Jansatta and Loksatta news text; five domains; OCR-baseline WER about 36%) and evaluate eleven open-weight models from 3B to 32B under seven prompting conditions. Their proposed CharBM25 retrieves demonstrations by BM25 over character n-grams of OCR inputs, aiming to match shared error patterns. It beats domain-random selection by 2.8-4.0 WER points on Hindi and matches or beats dense retrieval for 10 of 11 models per language; Gemma-3-27B yields the largest reduction (55.0% WER Hindi, 33.3% Marathi). Gains are unreliable below 8B parameters, Marathi remains consistently harder than Hindi, and character trigrams win the n-gram ablation. Main limitation: the entire corpus uses synthetic OCR corruption from a single engine and font, so generalization to real scans is unverified; no statistical significance tests are reported.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 5
● low 0
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CharBM25, a character trigram BM25 retriever over OCR inputs, beats domain-random few-shot selection by 2.8-4.0 pp absolute WER on Hindi (2.9-3.8 pp on Marathi) and matches or exceeds dense retrieval for 10 of 11 models per language.
- Few-shot ICL gains are capacity-gated: models below 8B parameters do not reliably improve over the OCR baseline, and 3B models degrade more Marathi sentences than they improve.
- Gemma-3-27B with CharBM25-5 (n=3) achieves 55.0% relative WER reduction on Hindi and 33.3% on Marathi against a ~36% OCR baseline.
- The Hindi-Marathi correction gap widens with scale (8.8 pp at 3B to 21.7 pp at 27B), attributed to Marathi's greater morphological complexity and named-entity density.
- This is the first systematic evaluation of prompted LLMs for Devanagari post-OCR correction.
Strengths
Comprehensive, fully-tabulated evaluation
11 open-weight models (3B-32B) x 7 prompting conditions x 2 languages, with per-domain WER (Table 5) and CER (Table 6) plus an n-gram ablation over n in {1,2,3} for every model (Tables 3-4) - unusual reporting depth for an ICL study.
Well-motivated retrieval design
Section 3 grounds CharBM25 in the orthographic nature of OCR errors (matra substitutions, conjunct fragmentation, Tables 1-2), and Section 3.1 honestly concedes dense retrieval captures similar overlap implicitly, positioning the contribution as cost (sub-millisecond CPU, no embedder).
Beyond aggregate metrics
Honest negative results
Public dataset
Weaknesses
No statistical testing
Synthetic-only OCR noise
Thin dense-retrieval baseline
Aya-101 framing inconsistency
Near-duplicate retrieval unexamined
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper claims two things: (1) the first systematic evaluation of in-context learning (ICL) for Devanagari post-OCR correction, across Hindi and Marathi, and (2) CharBM25, a character-trigram BM25 retriever over OCR inputs for few-shot demonstration selection. The related-work survey (Sections 2.1-2.3) plausibly supports the novelty claim - prior Indic work is fine-tuning-based (e.g., Bhandari & Harit 2026, RoundTripOCR) and prior LLM post-OCR work is European-language focused (Thomas et al., Kanerva et al., Boros et al.). The method itself is a straightforward adaptation of surface-form example selection (the authors correctly cite Ye et al. 2024 and Su et al. 2023 as token-level precedents), so the contribution is a well-executed empirical study rather than an algorithmic advance - and Section 3.1 is refreshingly candid that CharBM25's edge over dense retrieval is cost, not signal quality.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CharBM25 and how does it work?
CharBM25 is a retrieval method for choosing few-shot examples in LLM prompting. It scores candidate example sentences using classic BM25, but over character n-grams (trigrams by default) extracted from the OCR-corrupted input rather than over words. The idea is that OCR errors are orthographic, so examples whose OCR side shares the same character-level corruption patterns (e.g., the same matra or glyph confusions) make the most instructive demonstrations. It needs no GPU or embedding model and retrieves in under a millisecond per query.
How much does the paper improve OCR quality for Hindi and Marathi?
With the largest tested model (Gemma-3-27B) and 5 retrieved examples, relative word error rate (WER) falls by 55.0% for Hindi (from a 36.06% baseline to 16.22%) and by 33.3% for Marathi (from 35.11% to 23.42%). Smaller models help much less, and models below about 8B parameters often fail to improve over the raw OCR output, especially on Marathi.
Is this evaluation based on real scanned documents?
No. The benchmark is synthetic: clean news sentences are rendered as images (single font, Augraphy degradations simulating scan artifacts) and passed through Tesseract 5 OCR. This yields controlled paired data but means real-scan generalization, acknowledged by the authors as future work, remains untested.
Is the dataset from this paper publicly available?
Yes. The authors state the 20,000-sentence benchmark (Hindi and Marathi, five news domains, with the shot bank) is released on Hugging Face under 'AbhishekBhandari/Devanagari-OCR-ICL-Benchmark'. No separate code release is mentioned in the paper, though CharBM25 is a short, standard BM25 implementation over character trigrams.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1, 3-5 are text-extracted axis labels and could not be visually inspected; their underlying values are corroborated by Tables 3-6.
- The PDF extraction shows duplicated figure numbers (two 'Figure 3' and two 'Figure 4' captions), likely a source typesetting error.
- The abstract as provided is truncated mid-sentence at 'general-pur'; judgments about the full abstract rely on the body text.
- Sampling/decoding parameters are described only as 'default', so greedy-vs-sampled decoding could not be confirmed from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.