ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Experts
Dang Hoai Nam, Nguyen Duy Hieu, Quang Huu Hieu, Vo Nguyen Le Duy
cs.CV · cs.LG
Quick verdict
Unified multitask+sparse-MoE page-level HTR: best IAM CER and sound ablations, but trails specialized HTR on 4/7 datasets and reports no significance tests.
Summary
ExpertHTR is a unified framework for page-level handwritten text recognition across heterogeneous collections. Existing structural annotations are converted into a common Page–Region–Line representation and used to derive four tasks—Page Transcription, Physical Line Counting, Text-to-Line Localization, and Line-Span Transcription—without new manual labels. Starting from Qwen3.5-0.8B-Base, the authors replace six decoder FFNs with shared-expert sparse MoE layers (four routed full-MLP experts), using Sparsegen routing whose active expert count varies per token, plus sparsity and load-balancing regularizers. On seven benchmarks (BRESSAY, Bentham, HWDB2.0, IAM, READ-2016, RIMES, ScribbleLens), matched-update multitask training beats Page-Transcription-only training on all seven; dense multi-source training improves six of seven; and regularized Sparse MoE beats the dense baseline on six of seven (HWDB2.0 excepted). The final model outperforms evaluated zero-shot OCR/VLM checkpoints on six of seven datasets (Bentham excepted) and reports 2.92% CER on IAM paragraphs, claimed as state-of-the-art in that configuration. However, specialized segmentation-free systems (DAN family) remain far better on BRESSAY, READ-2016, RIMES, and ScribbleLens. Limitations include a capacity confound (1117.79M vs 852.74M parameters), unmatched training exposure in the multi-source comparison, single-run results without significance testing, and code only promised for release.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Organizing existing structural annotations via a Page–Region–Line representation into four tasks (Page Transcription, Physical Line Counting, Text-to-Line Localization, Line-Span Transcription) improves page-level HTR over Page-Transcription-only training on all seven datasets under matched optimizer-update budgets.
- A shared-expert sparse MoE using Sparsegen routing with sparsity and load-balancing regularization improves over the Dense MTL baseline on six of seven benchmarks (HWDB2.0 excepted).
- The unified single checkpoint outperforms evaluated zero-shot general-purpose OCR/VLM systems on six of seven datasets and achieves 2.92% CER, claimed state-of-the-art, on the IAM paragraph-level benchmark.
- Specialized segmentation-free HTR systems remain stronger on BRESSAY, READ-2016, RIMES, and ScribbleLens, so the unified model complements rather than replaces specialized recognizers.
Strengths
Matched-budget supervision ablation is a genuine control
Table 4 compares Page-Transcription-only vs single-source multitask models under identical optimizer-update budgets (Sec 4.1), so gains on all seven datasets (e.g., READ-2016 CER 190.46→134.06, Bentham 37.71→26.91) cannot be explained by extra compute — a fairness control many MTL papers omit.
Routing design is analyzed, not just asserted
Table 6 plus Figs 4–5 show Fixed Top-2 loses to Dense MTL on 5/7 datasets while regularized Sparsegen improves 6/7 over unregularized Sparsegen; layer-wise support size and load-balance statistics substantiate the claim that support varies by depth and source rather than acting as a de facto Top-k.
Unusually candid limitation accounting
Error decomposition reveals heterogeneous mechanisms
Evidence that auxiliary tasks interact
Weaknesses
Large residual gap to specialized HTR on 4/7 benchmarks
No variance, seeds, or significance testing
Capacity and compute costs never isolated or reported
Headline OCR/VLM comparison is zero-shot vs fine-tuned
Auxiliary-task ablation confined to one dataset
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
ExpertHTR — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ExpertHTR and what problem does it solve?
ExpertHTR is a single vision-language model trained to transcribe full pages of handwriting across seven public datasets in different languages and scripts (Portuguese, English, Chinese, German, French, Dutch). It reuses the structural annotations those datasets already contain (pages, regions, lines) to create four training tasks without new manual labeling, and adds sparse Mixture-of-Experts layers so the model's effective capacity can vary with each input.
What accuracy does ExpertHTR achieve?
The paper reports character error rates of 2.92% on IAM paragraph-level recognition (its best result, claimed state-of-the-art for that benchmark configuration), 4.63% on HWDB2.0, 5.59% on RIMES, and roughly 17.57% to 52.23% on BRESSAY, Bentham, READ-2016, and ScribbleLens, where specialized DAN-family systems remain substantially more accurate.
Does ExpertHTR beat existing HTR systems?
It outperforms the evaluated general-purpose zero-shot OCR and vision-language models (LightOnOCR, PaddleOCR-VL, DeepSeek-OCR, CHURRO) on six of seven benchmarks, with Bentham as the exception, but specialized page-level systems (DAN, Faster DAN, Meta-DAN) still achieve much lower error on BRESSAY, READ-2016, RIMES, and ScribbleLens.
Is the ExpertHTR code available?
The paper only promises a release, stating that the official implementation and experimental configurations will be made publicly available at github.com/DAIR-Group/ExpertHTR. All seven datasets used are publicly available, and hyperparameters are described in enough detail to support reimplementation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure interiors (Figs. 1–6) are raster images whose plot-level details cannot be extracted from the text; assessments of the routing and error analyses rely on captions and the numeric values quoted in the body text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.