MoirfEolas and CríochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphology
Jane Adkins, Abigail Walsh, Brian Davis, Elaine Uí Dhonnchadha
cs.CL
Quick verdict
MoirfEolas dataset and CríochScore metric measure tokenizer alignment with Irish morphology; Unigram-8k best (40.55%); useful but purely intrinsic evaluation.
Summary
The paper introduces two resources for evaluating how well subword tokenizers align with Irish morphology: MoirfEolas, a dataset of over 35,000 Irish words annotated with eclipses, prefixes, and suffixes, built by greedy-longest-matching handcrafted affix lists plus UniMorph entries against the Irish side of ParaCrawl V9 (1.45M words), and CríochScore, an intrinsic metric scoring 1/0.5/0 depending on whether each affix is cleanly, partially, or not segmented from the stem. The authors train six HuggingFace tokenizers (Unigram LM, SentencePiece Unigram, WordPiece, BPE, SentencePiece BPE, byte-level BPE) at 8k–64k vocabulary sizes. Best alignment is Unigram LM at 8k (40.55%), marginally ahead of SentencePiece-BPE at 16k (40.39%), while vanilla BPE at 64k is worst (9.6%); smaller vocabularies align better across all algorithms. Trade-offs emerge: Unigram's superior alignment comes with the worst compression (highest corpus token count) and lower Rényi efficiency. Absorption analysis shows eclipses are almost always merged into following characters. Main limitations: purely intrinsic evaluation with no downstream validation, no variance reporting across training runs (top-two gap is 0.16 points), a degenerate eclipsis-only score (constant 50%, based on 13 words split at punctuation), and no inter-annotator validation of the greedy-match annotations. Code is released on GitHub.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The Unigram Language Model algorithm aligns with Irish morphological boundaries more often than BPE, WordPiece and their SentencePiece/byte-level variants, peaking at smaller vocabulary sizes (Unigram-8k, 40.55% CríochScore).
- Morphological alignment of tokenization trades off against compression (Unigram has the highest corpus token count) and vocabulary efficiency (lower Rényi efficiency).
- The MoirfEolas construction pipeline (curated affix lists + UniMorph + greedy-longest-match against a web corpus, with FST-based cleaning) can be emulated by other morphologically rich languages.
- MorphScore is unsuitable for Irish due to flawed lemma→stem derivation and its 3-subword limit, as documented with nine worked examples in Appendix A.3.
Strengths
Genuinely new resource for an under-resourced language
MoirfEolas maps 35,000+ Irish words to eclipses, prefixes and suffixes (216/178/13 unique), built from ParaCrawl V9 (1,447,047 tokens) with curated lists (330 prefixes, 213 suffixes, 13 eclipses after dedup) — a scale 13x beyond MorphScore's 2,576 Irish entries (§3, A.3).
Transparent, honest reporting
The authors investigate the anomalous constant-50% CríochE column themselves (§6.1: 13 words with hyphen/apostrophe eclipsis pairs split at punctuation), explicitly defer downstream claims, and acknowledge the conflicting compression literature (Schmidt et al. 2024 vs Goldman et al. 2024).
CríochScore is more granular than MorphScore for Irish
Practically useful trade-off analysis
High reproducibility
Weaknesses
No statistical uncertainty anywhere
Eclipsis evaluation is degenerate
No annotation-quality audit of the retained dataset
Only from-scratch tokenizers evaluated
CríochScore weights are unargued
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a non-archival workshop poster (TokShop @ COLM 2026) presenting two artifacts for Irish NLP: **MoirfEolas**, a dataset mapping 35,000+ Irish words to their eclipses, prefixes and suffixes (216 unique prefixes, 178 unique suffixes, 13 eclipses), and **CríochScore**, an intrinsic metric scoring how well tokenizer segmentations respect those boundaries (1 = affix cleanly separated from the stem; 0.5 = separated but further fragmented; 0 = not separated; word score normalized over ≤3 components). Central claims: (1) Unigram LM tokenizers align with Irish morphology more often than BPE/WordPiece variants; (2) alignment trades off against compression (corpus token count) and Rényi efficiency; (3) the construction pipeline is emulatable for other languages. The paper positions itself against MorphScore (Arnett & Bergen 2025), rejected for Irish on data-quality grounds detailed in Appendix A.3, and adapts Suffix Recall/Precision (Erkaya 2022).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the MoirfEolas dataset?
MoirfEolas is a dataset of over 35,000 Irish words, each mapped to its eclipses (initial-mutation consonants), prefixes and suffixes — 216 unique prefixes, 178 unique suffixes and 13 eclipses. It was built by greedy-longest-matching curated affix lists plus UniMorph entries against the Irish side of ParaCrawl V9, and is designed to test whether tokenizers respect Irish morphological boundaries.
What is CríochScore and how is it calculated?
CríochScore is an intrinsic evaluation metric for tokenizers: for each annotated affix (eclipsis, prefix or suffix) a word gets 1 if the affix is cleanly separated from the stem as one token, 0.5 if separated but further split, and 0 if not separated; word scores are averaged into the tokenizer's overall score. The best system in the paper, Unigram LM with an 8k vocabulary, scores 40.55%.
Which tokenizer works best for Irish morphology?
In this paper, Unigram LM with an 8k vocabulary aligned best with Irish morphological boundaries (40.55%), narrowly ahead of SentencePiece-BPE at 16k (40.39%); vanilla BPE at 64k was worst (9.6%). Smaller vocabularies aligned better for every algorithm, but better alignment came with worse compression and lower Rényi efficiency.
Does morphological alignment of tokenizers improve model performance?
This paper does not test that. It is a purely intrinsic evaluation, and the authors state that downstream evaluation — for example machine translation or language modeling — is necessary to verify whether CríochScore alignment translates into better performance.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction mangles accented characters throughout (e.g., 'Cr´ıochScore'); cosmetic only — all tables and numbers were interpretable.
- Tables 1, 2 and 4 were extracted as dense space-separated blocks; column values were reconstructed from the stated header order and cross-checked against the prose, but not verified cell-by-cell against the PDF.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.