From Zero to Hero: An Open LLM Ecosystem for Armenian
Erik Arakelyan, Khatun Avetisyan, Meri Davtyan, Heghine Grigoryan, Nane Khachatryan, Hayk Shahsuvaryan, Henrik Sergoyan, Vahan Martirosyan
cs.LG · cs.CL
Quick verdict
First fully open, reproducible Armenian LLM ecosystem; excellent data hygiene, but single-run training and MT-derived benchmarks temper the headline claims.
Summary
This paper assembles an open LLM ecosystem for Armenian, a low-resource language: ArmWeb, a 4.37M-document news corpus (3.3B Gemma tokens) from a documented 15-year crawl with syndication-aware deduplication, hard leakage gates, and 13-gram decontamination against ten Armenian benchmarks; and ArmSTEM, 373K parallel English-Armenian math/science problems machine-translated with placeholder masking and verified by blind re-solving with exact answer match, plus a 300-item human evaluation (299 valid, perfect agreement). Continued pretraining of Gemma-4-E4B on a 69/6/20/5 news/STEM/replay/code mixture yields arm-gemma-e4b, scoring 0.500 mean accuracy on a six-task Armenian suite versus 0.477 for the unadapted base and 0.35–0.47 for existing open Armenian models. Ablations show news-only CPT causes catastrophic forgetting (−21.2pp on Belebele at lr 1e-4), a gentler learning rate recovers two-thirds, and swapping ~6% of news for verified translated STEM reverses the loss, ending +2.2pp above base. Contamination scans find 7.9–17.4% document-level overlap between public Armenian crawl slices and evaluation panels, including FineWeb-2's own train/test leak. Main limitations: each CPT configuration is a single run without seed variance; two of six suite tasks sit at chance and three are machine-translated benchmarks; the corpus is news-concentrated with limited register diversity.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ArmWeb (4.37M documents, 3.3B Gemma tokens) and ArmSTEM (372,907 verified EN–HY pairs) constitute the first open Armenian LLM training resources released with complete data and recipe.
- arm-gemma-e4b outperforms every existing open Armenian model (suite means 0.35–0.47) and its own unadapted base (0.477), reaching 0.500 mean accuracy on the six-task suite.
- News-only continued pretraining trades knowledge for fluency (−21.2pp Belebele at lr 1e-4), and swapping ~6% of the mixture for verified translated STEM data reverses the loss, ending +2.2pp above base.
- The three largest public Armenian crawl slices overlap evaluation panels at 7.9–17.4% of documents, including FineWeb-2's own train/test self-overlap, so perplexity-style evaluation of crawl-trained models is systematically inflated.
Strengths
Leakage and split hygiene run as hard engineering gates, not asides
Deduplication runs globally before splitting; three post-split checks enforce zero exact cross-split collisions and near-duplicate rates of 0.045%/0.010%/0.005% (val/iid-test/temporal-tail), with release builds aborting on violation (§2.1, Appendix F). The dedup engine is validated against datasketch (95.8% recall on a 1% sample) and NeMo-Curator (99.2% document-level agreement, Appendix E), with a Russian sister-collection control (0.9% vs 22.4% removal) showing duplication is a property of the Armenian news ecosystem.
Functional, not judgmental, verification of translated STEM data
ArmSTEM's Gate G2 blind-re-solves each Armenian problem (o4-mini) with exact answer match, after placeholder-masked translation; solver-limited items are honestly tagged (10.9% of accepted math, 27.6% of science, §2.2). Translator choice was pre-registered (switch to GPT-5.5 only if ≥5 points better; measured gap 2.5), and a 300-item human evaluation found 299/300 valid with κ=1.0.
Single-variable causal design for the forgetting finding
Scale-transfer of ablation conclusions is tested, not assumed
A reusable community finding on Armenian benchmark contamination
Weaknesses
One training run per configuration; training variance never estimated
Discriminative signal rests on three tasks, partially machine-translated
Corpus concentration limits ecosystem breadth
STEM-CPT-full confounds repetition with composition
Small, uncorrected generative-benchmark comparisons
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper claims five deliverables (§1): (1) ArmWeb, a 4.37M-document, 3.3B-Gemma-token Armenian news corpus; (2) ArmSTEM, 373K verified parallel EN–HY STEM problems; (3) arm-gemma-e4b, presented as the best open Armenian model and the first released with complete data and recipe; (4) an adaptation finding that verified translated STEM data reverses news-CPT's fluency-for-knowledge trade; (5) a contamination audit of the three largest public Armenian crawl slices. This is a language-ecosystem resource paper with empirical ablations; superlative claims are consistently hedged with 'to our knowledge', and negative results are disclosed in Appendix A.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is arm-gemma-e4b?
An open Armenian language model created by continued pretraining of Gemma-4-E4B on the ArmWeb news corpus and ArmSTEM translated STEM data (10B tokens total). It scores 0.500 mean accuracy on the authors' six-task Armenian evaluation suite versus 0.477 for the unadapted base, and is released with its full training data and recipe via the COPA-AI Hugging Face organization.
How much Armenian training data was released with this paper?
Two datasets: ArmWeb, 4.37 million news documents (about 3.3B Gemma tokens) crawled from 19 Armenian outlets over 15 years, and ArmSTEM, 372,907 verified English-Armenian parallel math and science problems (about 311M Armenian Gemma tokens) machine-translated and answer-verified, plus a 300-item human evaluation.
Why do low-resource language models lose knowledge after continued pretraining?
This paper shows news-only continued pretraining improves fluency while eroding knowledge — classic catastrophic forgetting (Belebele drops 21.2 points at learning rate 1e-4, and the existing HyGPT-10b shows the same pattern). English replay alone does not prevent it, but swapping about 6% of the mixture for verified translated STEM data reverses the loss, ending 2.2 points above the unadapted base.
Are Armenian benchmarks contaminated in public training corpora?
The paper's 13-gram scan finds 7.9% (CulturaX-hy), 10.9% (HPLT-v2-hy), and 17.4% (FineWeb-2-hy) of documents sharing an n-gram with ten Armenian evaluation sets, including FineWeb-2's own Armenian test split appearing in its training split — meaning perplexity-style evaluations of models trained on these crawls are inflated.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction appears complete through the references and Appendices A–G; no truncation detected, and all quoted numbers were verified against the surrounding table captions.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.