Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM
Adam Zachary Wasserman, David Beauchemin
cs.CL
Quick verdict
A French-only BabyLM model scores 85.97% on native QFrBLiMP with unusually honest ablations, but headline comparisons are cross-benchmark and several effects sit near its own noise floor.
Summary
The paper presents METRON-FR, a 125M-parameter GPT-2 pretrained for five epochs on 92.47M words of French (CHILDES, BabyBabelLM-FR, filtered Wikipedia), submitted to the BabyLM 2026 Strict multilingual track, with sentence oversampling weighted by a Haitian-Creole cognate lexical oracle. It reports 85.97+/-0.17% on QFrBLiMP (native Quebec-French minimal pairs), 69.68% accuracy / MCC 0.13 on fine-tuned QFrCoLA, 76.28% on translated BLiMP, and 62.80% on the BabyLM-weighted leaderboard. A cross-lingual GLUE grid (French-translated task data, rank-16 LoRA) shows gains on relational tasks (RTE +5.13pp) but regression on MNLI (-11.10pp). Bilingual Lexicon Induction aligns the model's embeddings to GPT-2 at p@1=68.84+/-8.61% versus 25% for a grammatically failed matched-size English model, argued to show alignment tracks competence rather than data volume. Forensic ablations show a 16K-tokenizer swap alone reproduces a 7.7pp GLUE collapse, a placebo control nullifies dictionary-axiom prompting gains, the translated BLiMP Supplement scores below chance due to lexical-frequency confounds, and EWoK stays at chance across five interventions. Main limitations: the primary benchmark is co-authored by one of the authors, several deltas sit within the authors' own ~4.9pp rebuild noise floor, the BLI test set has only 48 held-out pairs, and the symmetric English-model control for the GLUE protocol is missing.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A French-only 125M GPT-2 trained on 92.47M words reaches 85.97% on native QFrBLiMP, exceeding the best native-BLiMP scores posted for the other 2026 BabyLM official languages on their respective benchmarks.
- At the 125M/100M-word scale, single-token zero-shot scoring is dominated by tokenizer and answer-template artifacts, so tokenizer-swap sensitivity and placebo-controlled prompting should be standard reported diagnostics.
- Native Quebec-French evaluation outperforms translated BLiMP by 9.69pp, showing translated minimal-pair benchmarks confound competence with translation quality and lexical-frequency artifacts.
- BLI alignment to GPT-2 (p@1 = 68.84+/-8.61%) versus a grammatically failed matched-size English model (25.00%) shows cross-lingual structural alignment tracks acquired grammatical competence rather than training duration.
- Cross-lingual GLUE adaptation via French-translated task data plus rank-16 LoRA helps relational tasks (BoolQ, RTE, MRPC) but not world-knowledge or discourse tasks (MNLI, WSC).
Strengths
Systematic negative-result reporting
Section 5 and Table 14 report EWoK at chance across five interventions, Table 5 shows the translation-specific axiom effect is -0.27pp (placebo >= targeted), and the translated BLiMP Supplement's 34% below-chance score is explained by a lexical-frequency confound (Section 5.3) rather than spun as a result.
Clean single-intervention causal design for the tokenizer ablation
Table 4's v3a-v3d grid holds corpus, architecture, optimizer, and seed constant and shows only the 16K CDS tokenizer swap (v3d: 43.50%) reproduces the v2 GLUE-axiomatic collapse (43.30%), while instruction data and EWoK-targeted Wikipedia individually do not.
Explicit noise-floor quantification
Strong reproducibility package
Well-argued native-language evaluation case
Weaknesses
Primary benchmark has an authorship overlap
Headline comparisons span incomparable instruments and budgets
Central effect sizes sit at or below the self-admitted noise floor
BLI statistical basis is thin
Unexplained variance structure and test-metric selection
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is METRON-FR?
METRON-FR is a 125M-parameter GPT-2 language model pretrained from scratch on 92.47 million words of French (child-directed speech, the French part of BabyBabelLM, and filtered French Wikipedia) for the BabyLM 2026 'Strict' challenge, which limits training data to 100M words. It was submitted as a French-only entry and scored 85.97% on the Quebec-French minimal-pairs benchmark QFrBLiMP.
How does a French-only BabyLM model compare to English BabyLM models?
The paper reports 62.80% on the BabyLM-weighted leaderboard, in the range of prior Strict-track winners, and 85.97% on QFrBLiMP, a Quebec-French benchmark. Direct comparison with English models is deliberately limited: the English benchmarks are scored on translated items for this model, and each language's BLiMP is a separate instrument, so cross-language scores are context, not like-for-like rankings.
Why does the paper argue single-token zero-shot scoring is unreliable at small scale?
Because its ablations show scores responding to artifacts rather than competence: swapping only the tokenizer dropped a GLUE-style score by about 7.7 percentage points, a mere reshuffle of the same corpus moved it by about 4.9 points, and prepending random (not translation-relevant) word-pair 'axioms' improved scores as much as relevant ones. The authors recommend reporting tokenizer-swap sensitivity and placebo-controlled prompting alongside any such benchmark score.
What is the Haitian-Creole lexical oracle in this paper?
It is a corpus-curation heuristic: French training sentences are oversampled (3x weight) when dense in French cognates of the 300 most frequent Haitian-Creole lemmas, on the theory that creole-contact survival flags load-bearing vocabulary. No Haitian-Creole text is trained on, and the authors state its effect is not isolated against an unweighted baseline, so it remains exploratory.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (e.g., Figure 1's accuracy-vs-loss trajectory) appear only as captions in the extracted text; plotted data could not be verified.
- Some appendix artifacts (per-seed tables, rebuild_seed_sweep.json contents) are referenced but not reproduced in the extract, limiting verification of variance claims.
- Table 2's chance row (3.80%) is inconsistent with the stated 48 held-out BLI pairs; flagged as a question rather than adjudicated, since it could be an extraction artifact or an unstated candidate-set definition.
- Claims about 2026 venues, leaderboards, and external baselines (e.g., Dutch/Chinese BLiMP scores) could not be checked without network access.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.