Beetle: A Bilingual Model Suite for Modelling Second-Language Processing
Suchir Salhan, Catherine Arnett, James Michaelov, Paula Buttery
cs.CL
Quick verdict
Releases 285 bilingual + 45 mono LMs with checkpoints; staged curricula best predict L2 reading times, but single seeds and L2-share confounds temper claims.
Summary
Beetle is an open-source framework for controlled bilingual language-model pretraining, in which tokeniser, target language, training budget, and exposure schedule can each be varied independently. The authors train and release 285 bilingual and 45 monolingual 125M-parameter models across 21 first languages (English always the second language), five exposure curricula (balanced, simultaneous, sequential, episodic-classroom, late), and three data scales (100M, 2B, 24B tokens), each with roughly 30 checkpoints. Evaluating on MECO second-language eye-tracking reading times, learner-error discrimination (BLiSS), structural priming, and BLiMP-style grammaticality, they find that staged curricula better predict L2 reading times than balanced exposure, especially at smaller data scales and for typologically closer language pairs, while balanced exposure wins on English grammaticality (BLiMP 61.6 vs 56.3 at 100M) and BLiSS rankings anti-correlate with MECO (Spearman ρ=−0.47). Small Beetle models match or beat much larger multilingual LLMs on reading-time alignment but not on BLiSS. Main limitations: a single random seed per configuration, curricula confounded with total L2 token share, and uncontrolled comparisons against industrial multilingual baselines.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Staged and temporally structured exposure curricula (B2/B3/B5) improve alignment with human L2 English reading times relative to balanced bilingual training (B1), with the largest gains at smaller data scales and for typologically closer language pairs.
- Curriculum effects are task-dependent: balanced exposure wins on English grammaticality (BLiMP 61.6 vs 56.3 at 100M tokens) while staged curricula win on reading-time prediction, and MECO vs BLiSS curriculum rankings anti-correlate (Spearman ρ=−0.47, p<0.001).
- 125M-parameter bilingual Beetle models match or exceed substantially larger massively multilingual LLMs on L2 reading-time alignment, but not on BLiSS learner-error discrimination.
- At 100M tokens every bilingual curriculum outperforms matched L1-monolingual controls on English BLiMP (mean 61.6 best vs 52.9), indicating cross-lingual transfer benefits for L2 grammar.
- Introducing L2 mid-training causes an abrupt, curriculum-aligned spike in L1 sentence NLL followed by partial recovery, with larger disruption for typologically distant pairs (Chinese, Hindi).
Strengths
Unprecedented controlled release at scale
285 bilingual + 45 monolingual 125M models over 21 L1s, five curricula (B1–B5) and three data scales (§4), each with ~30 log2-scheduled checkpoints that restart at phase boundaries — the authors claim the first checkpoints available for several of these languages. Tokenised training data is released in the exact order seen by each model.
Confound-aware experimental design
Architecture (PicoDecoder, §3) and L2 (English) held fixed; BPE tokenisers trained for equal compression across both languages, neutralising a known cross-lingual confound; curricula composed from explicit primitives with a clean factor table (App. C, Table 4); 13-gram decontamination against 19 evaluation datasets (App. B).
Multi-domain triangulation with honest negative results
Statistically careful secondary analyses
Transparent limitations section
Weaknesses
No seed-level variance for headline claims
Timing-vs-amount confound not resolved in the text
Uncontrolled baseline comparisons and thin controls
Unexplained contradiction with the inverted-U literature
Bookkeeping inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Beetle (arXiv 2609.22633) is a resource-plus-analysis paper from Cambridge (Salhan, Arnett, Michaelov, Buttery). It contributes (1) a modular pretraining framework in which tokeniser, target language, training budget and exposure schedule vary independently; (2) a released suite of 285 bilingual + 45 monolingual 125M-parameter models over 21 L1s (English always L2), five exposure curricula (B1 balanced 50:50; B2 simultaneous sigmoid at 50%; B3 sequential to 33:67; B4 episodic 80:20 with bursty L2; B5 late onset at 80% of training), and three data scales (100M, 2B, 24B tokens), ~30 checkpoints each; (3) an empirical study of how curricula shape alignment with human L2 processing. Central claims: staged curricula better predict L2 English reading times (MECO first-pass duration) than balanced training, most at smaller scales and typologically closer pairs; balanced exposure is best for grammaticality (BLiMP 61.6 vs 56.3 at 100M, §4.1); MECO and BLiSS curriculum rankings anti-correlate (ρ=−0.47, p<.001); small Beetle models match or exceed much larger multilingual LLMs on reading-time alignment but not on BLiSS (Fig. 5: baselines reach RP@0 ≈ 66–71).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Beetle model suite?
Beetle is an open-source set of 285 bilingual plus 45 monolingual language models (125M parameters each) trained under controlled conditions to study second-language acquisition: 21 first languages with English always as the second language, five exposure curricula (balanced, simultaneous, sequential, episodic-classroom, late), and 100M/2B/24B-token training budgets, with about 30 checkpoints released per model.
Did sequential (staged) language training beat balanced bilingual training?
For predicting second-language English reading times (MECO eye-tracking), yes — staged curricula (B2/B3/B5) aligned better with human reading behaviour, especially at smaller data scales and for typologically closer language pairs. But for grammaticality judgements (BLiMP), balanced training was best (61.6 vs 56.3 at 100M tokens), so the curriculum effect depends strongly on which behaviour is measured.
Are the Beetle models, data and code publicly available?
Yes. The paper states that all checkpoints (~30 per model, including gradients and activations for interpretability), tokenisers, decontaminated and pre-tokenised training data in the exact order seen by each model, evaluation scripts, and the CSV files behind every figure are publicly released via the project page (beetlelm.github.io).
What are the main limitations of the Beetle study?
Three stand out: only one random seed was run per configuration, so curriculum rankings carry no uncertainty estimates; the curricula differ in total amount of second-language data (50% down to 10%), so timing effects are partly entangled with exposure quantity; and comparisons against large multilingual models (Apertus, XGLM, Gemma, Qwen) are not controlled for tokeniser, data, or scale.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are not verifiable from the extracted text — including Figure 4, which carries the core MECO ∆logL evidence; exact values, error bars and axis scales could not be checked.
- Appendix extraction is partial: the exposure-ablation results (B1b, B4a/B4b) and parts of Tables 3–4 are described but not fully present, so claims that timing effects survive quantity controls could not be verified.
- Some table oddities may be extraction artefacts rather than paper errors (missing Human-scale zho–eng B1/B4 rows in Table 5; Table 2's 'Languages 9/20/20' vs 19 L1s listed in §4).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.