Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition
cs.CL
Quick verdict
Rigorous measurement study: Sanskrit's apparent token advantage is an artefact at 32k/64k under matched controls; honest scoping, primary bias sign unknown.
Summary
The paper asks whether Sanskrit's per-word information density survives subword tokenization, measuring cost per proposition rather than per word. On FLORES-200 devtest, deployed tokenizers with vocabularies of 200,019 ids or more charge Sanskrit 1.774–2.187× the English tokens (7.857× under GPT-2), but only 1.060–1.353× Hindi; fertility-based ratios would overstate all of these gaps. The methodological core is a matched control: BPE/Unigram tokenizers trained with the same algorithm and vocabulary size (32k/64k/128k) on the English side of the same corpora (Sāmayik prose, Itihāsa verse), in pair-matched and byte-matched forms. Against a deployed English pivot, a trained Sanskrit BPE looks cheaper per proposition (0.887); against the matched control this flips: at 32k and 64k all 8 matched pairs sit above 1.0 on prose (1.030–1.142, 95% paired bootstrap intervals excluding 1.0), and at 128k the in-domain value drops to 0.983 while staying above parity out of domain (1.025) and on FLORES (1.116). Exact factorization shows TPP = character-length ratio × tokens-per-character, the latter near 1; Sanskrit prose in SLP1 is not shorter than English (1.028). Verse stays below parity (0.555–0.658), but the authors decline to interpret it given meter and 19th-century translation confounds. Limitations: unknown translationese-bias sign on the primary prose corpus, only 11.21 MB of training data, no downstream validation, no multiplicity correction. Code, the results snapshot and pre-registration comparisons are public.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Deployed tokenizers with vocabularies of 200,019 ids or more charge Sanskrit 1.774–2.187× the English tokens on identical FLORES-200 content, and 1.831–2.899 English tokens per proposition on prose and FLORES under the deployed o200k pivot in SLP1 — the paper's robust deployed-practice finding.
- Fertility systematically overstates the Sanskrit token penalty because sandhi and compounding shrink the whitespace-word denominator (fertility-implied 2.31–8.38 Sa/En vs measured parity 1.774–7.857; 1.60–1.74 vs 1.060–1.353 Sa/Hi).
- The apparent below-parity cost of Sanskrit-trained BPE against a deployed English pivot (0.887 at 64k pieces) is an artefact of the pivot's domain handicap: against matched same-algorithm, same-vocabulary, same-corpus English controls, all 8 pairs at 32k/64k sit above 1.0 on prose with intervals excluding 1.0.
- Tokens per proposition factorises exactly into a character-length ratio and a tokens-per-character ratio, the latter near 1 throughout, so matched ratios mostly measure relative text length — and Sanskrit prose in SLP1 is not shorter than English (1.028).
- The controlled penalty shrinks with vocabulary size, crossing below parity only at 128k pieces on in-domain prose (0.983 [0.971, 0.997]) while staying above it out of domain (1.025) and on FLORES (1.116) — a result the authors scope explicitly to the sizes measured.
Strengths
Matched-control design removes a demonstrated confound
Same-algorithm, same-vocabulary English controls trained on the English side of the same corpus (E1 pair-matched, E1_bm byte-matched; Table 2) flip the headline 0.887 [0.875, 0.899] crossing of BPE-64k vs the deployed o200k pivot to 1.035 [1.021, 1.049]. The paper proves the entire gap is the English denominator (33,702 vs 39,339 control tokens on identical English text, a 14.3% domain handicap), since the Sanskrit numerator is shared by construction.
Dual-control scheme anticipates and closes counter-readings
Pair-matching leaves English with 48% more training bytes (16.55 MB vs 11.21 MB); the byte-matched family E1_bm closes that reading off — all 24 ratios move ≤0.025 (median 0.008), 23 downward, zero verdict changes (Section 6.3). Both controls are reported for every pair rather than cherry-picked.
Exact factorisation with a no-vocabulary reference
Reproducibility discipline unusual for a solo preprint
Fertility vs parity separation is a service to the field
Weaknesses
Unknown translationese sign on the primary corpus
Thin training corpus for the largest-vocabulary claims
No downstream anchor for the metric
Provenance gaps in the deployed arms
No multiplicity correction across dozens of intervals
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks one question: does Sanskrit's celebrated per-word information density survive subword tokenization when cost is counted per unit of meaning rather than per word? It carefully separates Claim A (Sanskrit is dense per word — cited via Briggs 1985, not tested) from Claim B (that density survives tokenization — tested). Three deliverables: the measurement (parity, Eq. 1, and tokens-per-proposition, Eq. 2, across 4 deployed general-purpose, 3 deployed Indic and 6 self-trained tokenizers); the matched English control family (E1 pair-matched, E1_bm byte-matched; BPE/Unigram at 32k/64k/128k pieces); and a public artifact repository. Headline numbers: deployed tokenizers with ≥200,019-id vocabularies charge Sanskrit 1.774–2.187× English tokens on FLORES-200 devtest (GPT-2, 50,257 ids: 7.857×; Table 4); 1.831–2.899 English tokens per proposition on prose and FLORES under the o200k pivot in SLP1 (Tables 5–8). The abstract's framing — the robust statement is about deployed practice — matches what the evidence supports.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How many more tokens does Sanskrit cost than English in LLM tokenizers?
Under tokenizers people actually deploy (o200k, Llama-4, Gemma-3, GPT-2), this paper measures Sanskrit at roughly 1.8–2.9 English tokens per proposition on contemporary prose and FLORES-200 content, with Sanskrit written in SLP1 romanisation. On identical FLORES sentences, parity ratios run 1.774–2.187 for tokenizers with vocabularies above 200,000 ids, and 7.857 under GPT-2, whose 50k vocabulary predates Devanagari coverage.
Is Sanskrit really more information-dense than English for language models?
At the word level, yes, but the word-level advantage does not become a token-level saving. Under a fair comparison — same BPE algorithm, same vocabulary size, trained on the English side of the same corpus — Sanskrit prose costs slightly more tokens per proposition than English at 32k and 64k vocabularies (all 8 matched pairs above 1.0), and only reaches 0.983 at 128k on in-domain text, staying above parity out of domain. What survives matched tokenization is character length, and Sanskrit prose in SLP1 is not shorter than its English translation (1.028).
Why is fertility a misleading metric for Sanskrit tokenization?
Fertility divides token counts by whitespace word count, but sandhi (obligatory fusion at word boundaries) and compounding make Sanskrit words fewer and longer, so the denominator is artificially small and the penalty looks larger than it is. The paper measures fertility-implied ratios of 2.31–8.38 (Sanskrit/English) versus true parity of 1.774–7.857, and 1.60–1.74 versus 1.060–1.353 for Hindi — read off fertility, the Hindi hypothesis would have been falsely confirmed for all four deployed tokenizers tested.
Does training a Sanskrit-specific tokenizer fix the token penalty?
Not at the vocabulary sizes tested with raw subword training. A BPE trained on raw sandhied Sanskrit looks cheaper than English (0.887 at 64k pieces), but only against a deployed English tokenizer whose domain handicap explains the whole gap. Against matched English controls the advantage disappears at 32k/64k; only at 128k pieces does in-domain prose cross parity (0.983), staying above it out of domain and on FLORES. Sandhi-splitting or morphology-aware tokenizers remain untested and are named as follow-up work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 appear only as extracted axis labels, legend entries and captions; visual content (e.g., curve shapes in Figure 2) was assessed from captions and the corresponding tables, not from rendered plots.
- The abstract in the supplied metadata is cut mid-sentence, but the complete abstract is present at the head of the full paper text, so no paper content was lost.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.