A Grapheme-Aware Indic Tokenizer for Tamil: Large-Scale Training and Intrinsic Evaluation
Hari Krishnan K, Sudarsun Santhiappan
cs.CL
Quick verdict
Grapheme-mapped WordPiece for Tamil tops 5 multilingual tokenizers on intrinsic metrics, but lacks a no-mapping control and any downstream validation.
Summary
The paper introduces a grapheme-aware WordPiece tokenizer for Tamil. A preprocessing stage normalizes Unicode text, extracts grapheme clusters, and maps each cluster to a reversible atomic symbol before WordPiece vocabulary learning, so subword units never split graphemes. The tokenizer was trained on a cleaned Tamil Wikipedia corpus (569,055 articles, ~42.8M words, 75.4% Tamil characters) at four vocabulary sizes (10K-50K) and evaluated intrinsically on 5,000 FLORES+ Tamil samples against GPT-2, mBERT, mT5, mBART, and NLLB, using compression ratio, fertility, characters-per-token, word fragmentation rate, percentage of pretokens changed, normalized sequence length, and vocabulary coverage. The 50K model achieves fertility 1.55, CPT 5.83, WFR 0.388, and NSL 0.063 (vs GPT-2), all best among the compared tokenizers, while matching mT5's compression ratio (9.0571); a qualitative example yields 10 tokens versus 19 for mBERT and 157 for GPT-2. Main limitations, largely acknowledged by the authors: evaluation is intrinsic only, with no language-model or downstream validation; there is no control tokenizer - WordPiece trained on the same Tamil corpus without grapheme mapping - so gains cannot be causally attributed to grapheme awareness rather than language-specific training; recently proposed Indic-specific tokenizers are excluded; the training corpus is Wikipedia-only; and no statistical significance testing is reported.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Mapping complete Tamil grapheme clusters to reversible atomic symbols before WordPiece learning preserves grapheme integrity while remaining fully compatible with standard transformer tokenization pipelines (§3).
- Increasing vocabulary size from 10K to 50K monotonically improves intrinsic segmentation quality - fertility falls 1.9841→1.5527, CPT rises 4.5649→5.8332 - while coverage stays constant at ~99.96% (§6.1-6.2, Table 4).
- The 50K grapheme-aware tokenizer achieves the best fertility (1.5527), CPT (5.8332), WFR (0.3884), PPC (0.3884), and NSL (0.0631) among the six compared tokenizers on 5,000 FLORES+ Tamil samples (Table 5).
- Despite a language-specific vocabulary, the proposed tokenizer matches the highest observed compression ratio (9.0571, equal to mT5) (§6.5).
Strengths
Simple, deterministic, plug-compatible design
The reversible grapheme-to-symbol mapping (§3.2-3.4, Algorithm 1) preserves grapheme integrity with a bijective map and requires no modification to existing WordPiece implementations, making adoption trivial for standard transformer pipelines.
Clean vocabulary-scaling study
Only vocabulary size varies across the four runs (10K/20K/30K/50K, §4.1); Table 4 shows monotone fertility decrease 1.9841→1.5527 and CPT increase 4.5649→5.8332, isolating the vocabulary-size effect.
Substantial, disclosed training corpus
Metric honesty
Large margins on the informative metrics
Weaknesses
No no-mapping control ablation
Intrinsic-only evaluation
Two of seven metrics are non-discriminative as computed
Confound-laden baseline comparison
Thin scholarship and unstated details
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This preprint (arXiv 2609.06690, September 2026) proposes a grapheme-aware WordPiece tokenizer for Tamil. The mechanism (§3): normalize Unicode text, extract grapheme clusters, bijectively map each cluster to an atomic symbol, then run unmodified WordPiece vocabulary learning over the mapped stream; decoding inverts the map. The paper claims (i) vocabulary learning over grapheme atoms yields linguistically meaningful subwords, (ii) vocabulary scaling from 10K to 50K monotonically improves intrinsic quality (Table 4), and (iii) the 50K model dominates five multilingual tokenizers on fragmentation/efficiency metrics while matching the best compression ratio (Table 5). All evaluation is intrinsic; the authors say so plainly in §7.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a grapheme-aware tokenizer?
A tokenizer that first groups multi-code-point characters (in Tamil, a consonant plus vowel signs can span several Unicode code points) into complete grapheme clusters and maps each cluster to a single atomic symbol before learning the subword vocabulary, so learned subwords never split a visible character.
How much shorter are token sequences for Tamil compared to GPT-2 or mBERT?
On FLORES+ Tamil samples, the proposed 50K tokenizer's normalized sequence length is 0.0631 relative to GPT-2 (roughly 16x shorter) with fertility 1.55 tokens per word versus mBERT's 3.59; in the paper's example sentence it produced 10 tokens against 19 for mBERT and 157 for GPT-2.
Does the tokenizer improve downstream model performance?
This is unknown: the paper evaluates tokenization only intrinsically (compression, fragmentation, sequence length, coverage) and explicitly leaves downstream validation - training language models or translation systems with the tokenizer - to future work.
Is code or data available for this tokenizer?
The paper cites a GitHub implementation (github.com/sudarsun/indic-tokenizer) and a public Kaggle Tamil Wikipedia dataset; the method itself is standard WordPiece preceded by a reversible grapheme-to-symbol preprocessing map, so it is straightforward to reimplement.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Tamil strings in the extracted text (notably the Table 6 tokenizations) contain combining-mark rendering artifacts, so the qualitative example could not be verified at the character level; all quantitative tables extracted cleanly.
- Figure 1 is a diagram; only its caption survives text extraction, so the figure itself was not inspected (the §3 prose describes the same pipeline).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.