Combining Synthetic and Real Data for Low-Resource Historical OCR: A Manchu Case Study
Yan Hon Michael Chung, Hanlin Wang
cs.LG
Quick verdict
It claims real data lifts Manchu OCR to 95.09–96.28% WA (voting 98.27%); a rigorous, transparent study, though single-run and on a 753-image benchmark.
Summary
This empirical study examines how synthetic and real training data should be combined for OCR of a low-resource historical language, using Manchu as the case. The authors fine-tune three pretrained vision-language models (LLaMA-3.2-11B-Vision, Pixtral-12B, Qwen3-VL-8B) and train a compact 12-million-parameter CRNN under four regimes — synthetic-only, real-only, joint, and sequential synthetic-to-real — using 60,000 synthetic and 20,306 real word images, with checkpoint selection on a held-out validation split and final evaluation on a fixed 753-image archival benchmark. Adding real images lifts the leading configurations to 95.09-96.28% word accuracy against a synthetic-only ceiling of 87.92%; joint and sequential training perform comparably; and the small CRNN matches the best VLMs once real data are available. Because errors of the three strongest recognizers are largely complementary, a voting ensemble with dictionary-guided adjudication reaches 98.27%, though plain majority voting scores identically. Main limitations: each configuration is a single training run with uncontrolled CRNN seeds; the test set is small and lexically overlaps training data; the joint-versus-sequential comparison confounds training order with warm-start provenance; and the CRNN's response to synthetic data is sensitive to a corrected CTC objective studied only as a robustness check.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Introducing 20,306 real historical training images raises the leading configurations to 95.09-96.28% ARCH-test word accuracy, versus a synthetic-only ceiling of 87.92%.
- Joint and sequential synthetic-real training yield statistically indistinguishable archival accuracy under the tested practical pipelines (all paired 95% CIs include zero).
- A compact ~12M-parameter CRNN reaches the leading performance band once real images are available, showing model scale alone does not determine recognition accuracy.
- Voting over three complementary recognizers raises word accuracy to 98.27% without additional training, with an eighteenth-century Manchu dictionary providing a principled adjudication rule that, however, scores identically to plain majority voting on this benchmark.
Strengths
Systematic 4x4 comparison under a single protocol
16 model x regime configurations (Table 2) share one checkpoint-selection rule (SCI-val peak, deterministic tie-breaks, Section 4.3) and one untouched archival test set, making cross-model statements about training-data composition unusually clean for this literature.
Unusually transparent reporting of inconvenient results
Section 6.3 and Table B.2 admit the QWJ dictionary adds 0.00 pp over plain majority voting on ARCH-test (both 98.27%); footnote 1 dissects the CRNN-SYN 64.28% vs prior 56.8% gap into a checkpoint-selection effect (52.99% at a fixed epoch); Table B.3 exposes tied-peak spread (CRNN-JOINT alternates reach 96.15-96.41% vs selected 95.48%).
Paired statistics with honest nulls
Disclosed bug with controlled re-examination
Practical deployment relevance
Weaknesses
Single run per configuration, uncontrolled randomness
Small, lexically overlapping benchmark
JOINT vs SEQ is not a controlled comparison
No stated code or artifact release
Cross-model conclusions rest on one shared recipe
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed question: once annotated real images exist, how should they be combined with synthetic data for low-resource historical OCR? Using Manchu, the authors build a 16-cell grid (three VLMs plus a CRNN x four regimes: SYN, REAL, JOINT, SEQ) under one checkpoint-selection rule and one archival test set (ARCH-test, 753 images from Chung and Choi 2026). Four central claims: (1) real data lifts leading configurations to 95.09-96.28% word accuracy versus a synthetic-only ceiling of 87.92% (Table 2); (2) JOINT and SEQ perform comparably (every paired CI includes zero, Table B.2); (3) a ~12M-parameter CRNN joins the leading band once real data are available; (4) voting over LLaMA-SEQ/Pixtral-SEQ/CRNN-JOINT reaches 98.27%, with the QWJ dictionary adjudicating disagreements. All four claims are supported at the level the evidence allows, and the authors largely police their own wording (the leading band is 'descriptive rather than a claim of statistical equivalence', Section 5.1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How accurate is this Manchu OCR approach?
On a 753-image benchmark of real Qing manuscripts and prints, the best single models reach 95.09-96.28% word accuracy after training on real historical images (versus at most 87.92% with synthetic images only). A voting ensemble of three complementary models reaches 98.27%.
Is synthetic training data enough for low-resource OCR?
Not in this study: synthetic-only training capped at 87.92% word accuracy (best VLM) and 64.28% (CRNN). Adding about 20,000 real annotated word images was the decisive factor, lifting leading configurations by roughly 8-14 percentage points depending on the model.
Do you need a large vision-language model, or is a small model enough?
Once real training images are available, a compact 12-million-parameter CRNN matched the best fine-tuned VLMs (within 0.4 percentage points) and was over 100x faster (about 8 ms versus 1.5-2.3 seconds per word), so model scale alone did not determine accuracy.
What does the eighteenth-century Manchu dictionary contribute?
It supplies a principled, parameter-free rule for deciding among disagreeing recognizer outputs using word attestation. On this benchmark it did not raise accuracy beyond plain majority voting (both 98.27%), but it offers a reproducible adjudication mechanism for cases where no majority exists.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure graphics (Figures 1-6) are available only via captions and in-text descriptions in the extraction; plotted details such as error-bar extents in Figure 3 were not directly inspected, though all underlying numbers appear in Tables 2, B.1-B.3.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.