Objective vs. Search: Decomposing What Makes a Good Tokeniser
Ahmetcan Yavuz, Clara Meister, Tiago Pimentel
cs.CL · cs.AI
Quick verdict
Careful 2x2 study showing search procedure—not objective—drives tokeniser bits-per-byte; well-controlled, though effects are small and top-down scoring remains approximate.
Summary
The paper disentangles two confounded design axes in subword tokeniser learning: the optimisation objective (compression vs. unigram log-likelihood) and the search procedure (bottom-up merging vs. top-down pruning). Since BPE and UnigramLM differ on both axes simultaneously, prior comparisons (e.g., Schmidt et al. 2024) cannot attribute performance differences to either. The authors complete the 2x2 design space with two new algorithms: BottomUpLL (BPE-style greedy merging scored by an exact unigram log-likelihood gain, Lemma 2) and TopDownComp (UnigramLM-style pruning scored by compression, Lemma 4). They train Llama-style LMs (100M-1B parameters, Chinchilla-style 20-tokens/param budgets) with all four tokenisers across 8k/32k/128k vocabularies in English (FineWeb-Edu) and a five-language multilingual setting, evaluating bits-per-byte (BPB) and minimal-pair grammaticality. The headline result: search procedure dominates-BPb bottom-up tokenisers achieve lower BPB in nearly all conditions (only exception: 300M/32k, delta 0.0003), with paired document-level bootstrap CIs excluding zero; BLiMP-family accuracy shows no consistent pattern. Surprisingly, at 32k/128k bottom-up tokenisers win on both intrinsic compression and log-likelihood regardless of objective, and vocabularies cluster by search procedure (80.5% BPE-BottomUpLL overlap). Key limitations: top-down deletion costs use a local-replacement approximation validated only at small scale (81.5% vocabulary overlap with exact scoring), single seeds for 300M/500M/multilingual runs, BottomUpLL's high seed variance at 1B, and no downstream evaluation beyond BPB and grammar benchmarks.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Search procedure, not optimisation objective, is the dominant factor in bits-per-byte: bottom-up tokenisers (BPE, BottomUpLL) beat objective-matched top-down counterparts in nearly all model/vocabulary/domain settings (Tabs. 2-3, Tab. 9).
- At vocabulary sizes of 32k and above, bottom-up methods achieve both lower Gcomp and lower Gll than top-down methods regardless of their nominal objective, whereas at 8k the intrinsic metrics align with the objective as expected (Tab. 1).
- Minimal-pair grammaticality (BLiMP/MultiBLiMP/ZhoBLiMP) shows no consistent relationship with either design axis (Tabs. 2-3).
- The objective still shapes distributional properties: within each search family, likelihood-based tokenisers induce steeper Zipf slopes, lower entropy, and lower Coverage-50% (Tab. 1, Fig. 2).
- Learned vocabularies cluster primarily by search procedure: BPE-BottomUpLL overlap is 80.5% and TopDownComp-UnigramLM 63.6%, while cross-family overlap stays at 30-37% (Fig. 3, Tab. 6).
Strengths
Clean factorial decomposition of a genuine confound
Completing the 2x2 with BottomUpLL and TopDownComp under identical preprocessing (NFC + GPT-2 regex byte-level pretokenisation, Sec. 6/Appendix G) isolates each axis; this directly corrects the common misreading of Schmidt et al. (2024) as evidence that the compression objective itself is superior.
Unusual scale and breadth for a tokeniser ablation
Llama-style models up to 1B parameters at Chinchilla-style budgets, three vocabulary sizes, and both English-only (Tab. 2) and five-language multilingual (Tab. 3) conditions, with 3-seed replicates at 100M and English 1B/128k.
Statistical care beyond point estimates
Mechanistic analysis beyond aggregate scores
Weaknesses
Asymmetric exactness between the two search families
Small effect sizes relative to seed noise
Narrow evaluation surface
Under-specified top-down configuration
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the difference between BPE and UnigramLM tokenisers?
They differ on two independent axes at once: the objective optimised (BPE maximises corpus compression; UnigramLM maximises unigram log-likelihood) and the search procedure (BPE builds the vocabulary bottom-up via greedy merges; UnigramLM starts from an oversized vocabulary and prunes top-down). Because both axes change together, standard BPE-vs-UnigramLM comparisons cannot tell which one drives the observed differences.
Does the tokeniser's objective or its search algorithm matter more for language model quality?
In this paper's experiments the search procedure dominates bits-per-byte: bottom-up tokenisers (BPE and the new BottomUpLL) achieve lower bits-per-byte than top-down tokenisers (TopDownComp and UnigramLM) in nearly all tested settings, while the objective had no consistent effect on grammar benchmarks (BLiMP). At small vocabularies (8k) the objective still matters, with likelihood-based tokenisers doing better.
What are BottomUpLL and TopDownComp?
Two newly introduced tokeniser learning algorithms that complete the 2x2 design space spanned by objective and search procedure: BottomUpLL performs BPE-style bottom-up merging but scores each merge by its exact unigram log-likelihood gain; TopDownComp performs UnigramLM-style top-down pruning but scores each deletion by its compression cost.
Is a more compressed tokeniser always better for language modelling?
Not according to this paper: intrinsic compression did not uniquely predict model quality. Bottom-up search predicted low bits-per-byte even when the tokeniser's objective was likelihood rather than compression, and neither objective nor search showed a consistent relationship with grammatical accuracy on minimal-pair benchmarks.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (Fig. 2 rank-frequency curves, Fig. 4 five-set Venn) are available only as text descriptions; judgments about them rely on the reported tables and captions.
- Some tokeniser-training details (e.g., top-down seed vocabulary size |S0|) may live in appendix sections that are partially extracted; Sec. G as extracted covers preprocessing only.
- Review based on arXiv v1 (2609.19145, 16 Sep 2026); no peer-review record exists yet, and the code repository link (Ahmetcanyvz/comp-vs-like) was not verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.