KhatianDoc: A Human-Verified Benchmark Diagnosing Multimodal LLM Failure on Bengali Legal Land Records
Tasmiad Hasan, Arafat Zaman Ratul, Sarker Sadman Saalim, S. M. Shah Nawaz Hossain, Khan Raiyan Ibne Reza, Sumaiya Tabassum Nimi
cs.CL
Quick verdict
First human-verified benchmark for Bengali land records; zero-shot multimodal LLM failure well-evidenced, honest self-audit, but single-Mouza, no error bars.
Summary
This paper introduces KhatianDoc, a four-task benchmark built from 107 real RS Khatian land records from Munshiganj, Bangladesh, targeting Bengali legal document understanding and the Ana-Ganda-Kora-Kranti-Til base-16 fraction system. Ground truth was transcribed entirely by hand and independently verified by a land-law practitioner to full agreement, then anonymized with positional tokens that preserve the referential distinctions multi-hop questions need. The tasks cover symbol recognition (95 crops, 52 strings), fraction-to-decimal conversion, structured field extraction (261 rows across 107 documents), and QA over 1,634 pairs with a 300-question stratified evaluation subset. Six multimodal LLMs (8B to 72B+, accessed uniformly via OpenRouter under a fixed zero-shot protocol) are evaluated: five QA categories, 39.3% of the subset, return zero correct answers from every model, and on arithmetic every model's MAE (0.40-0.42) exceeds a constant-mean baseline (0.237), with exact and near match coinciding, indicating outputs decorrelated from the input. A metric self-audit corrects a refusal-scoring bug (Gemini 2.5 Flash Lite EM 13.33 to 21.67; Llama 4 Scout 11.33 to 18.33) and flags metadata EM as an inflated upper bound due to constant and null fields. Main limitations: a single-Mouza corpus, zero-shot-only evaluation without CoT, fine-tuning, or OCR augmentation, exact-string scoring of complex QA, and no statistical uncertainty reporting.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- KhatianDoc is the first benchmark for Bengali RS Khatian land records and the first machine-checkable, legally verified encoding of the Ana-Ganda base-16 fraction system.
- Five Task 4 QA categories totaling 39.3% of the 300-question stratified subset (118 questions) return zero correct answers from all six evaluated multimodal LLMs.
- On base-16 arithmetic, every model that emits a number underperforms a constant-mean baseline (MAE 0.40-0.42 vs 0.237), and Exact = Near Match for all models, indicating outputs decorrelated from the input.
- Two evaluation artifacts were found and handled transparently: a refusal-scoring bug (corrected scores reported beside originals for Gemini 2.5 Flash Lite and Llama 4 Scout) and an inflated metadata EM, demoted to an upper bound with row-level F1 as the Task 3 metric of record.
Strengths
Genuinely novel, legally motivated resource
§1-3.4: first benchmark for RS Khatian records and, per the authors, the first machine-checkable encoding of the Ana-Ganda base-16 fraction system (rules + worked example in Appendix B, e.g. 3 Ana 5 Ganda 2 Kora = 0.2046875); the gap is script/numeral-system invisibility, not just language coverage.
Rigorous ground-truth pipeline with no circularity
§3.2: every field hand-transcribed, no OCR at any point (deliberate, to avoid folding in the very errors being measured), then independently reviewed line-by-line by a land-law practitioner reaching complete agreement on glyph identity and decimal value.
Rare metric self-audit transparency
Privacy design that preserves task validity
Decorrelation argument beyond low scores
Weaknesses
Narrow corpus limits external validity
No uncertainty quantification anywhere
Narrowest possible evaluation condition
Exact-string scoring of complex legal QA
Model suite skews to small/cheap variants
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
KhatianDoc — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is KhatianDoc?
KhatianDoc is a four-task benchmark built from 107 real handwritten Bengali land records (RS Khatians) from Munshiganj, Bangladesh. It covers symbol recognition, base-16 fraction-to-decimal conversion, structured field extraction, and legal document QA over 1,634 pairs, all with ground truth transcribed by hand and verified by a land-law practitioner.
What is the Ana-Ganda-Kora-Kranti-Til numeral system?
It is Bangladesh's base-16 positional fraction system used in land records: 16 Ana = 1 plot, with subunits Ganda (20 per Ana), Kora, Kranti, and Til. Ana values use dedicated Unicode glyphs (U+09F4-U+09F9) that lack mainstream font and OCR support.
What did the paper find about multimodal LLMs on these documents?
Under a fixed zero-shot protocol, six multimodal LLMs (8B to 72B+) answered zero questions correctly in five of ten QA categories, which make up 39.3% of the 300-question evaluation subset, and every model did worse on fraction arithmetic than a baseline that always predicts the dataset mean (MAE 0.40-0.42 vs 0.237).
Is the KhatianDoc dataset publicly available?
Yes. Anonymized text and redacted page images (personal identifiers, header numbers, and location labels masked) are on Hugging Face (RaiyanKhaan/KhatianDoc). The original unredacted scans are access-controlled, and the paper's reported numbers were computed on those originals.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2's per-category values arrive in the extraction as a flat numeric stream; individual category-x-model cell values could not all be verified against an aligned table, though the Table 3 counts and totals cross-check.
- Appendix F in the provided text shows prompt templates for Tasks 2 and 4 only; whether Task 1/3 templates appear beyond the provided pages cannot be verified from the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.