A retrospective analysis on the use of LLMs to study infant syntax learning
Hélie Bazin, Anouk Barberousse, François Yvon
cs.CL
Quick verdict
BabyLM models stay below human syntax scores and CDS/TS effects look mixed; fair epistemological critique, though no new experiments and curated evidence.
Summary
This paper offers a philosophy-of-science assessment of the BabyLM challenge and related efforts that use LLMs trained on developmentally realistic corpora to study infant syntax acquisition. Rather than running new experiments, the authors survey how child-oriented datasets are assembled (transcribed-speech proportions range from 0-100% and child-directed speech from 0-58% across reviewed studies), which architectures are used (MLMs like BERT variants vs CLMs like GPT-2), and which benchmarks (BLiMP, Zorro, MSGS, PoSH-BENCH, HierQ, PG/ATB) evaluate syntax. A retrospective comparison of published scores yields four observations: models trained on 100M words or fewer rarely approach the ~90% human BLiMP baseline (best: GPT-BERT at 0.86 with 100M words); most learning happens below 10M words; MLMs outperform CLMs; and the amount of child-directed or transcribed speech in training data shows no consistent effect on benchmark performance. The authors conclude that LLMs neither refute nor prove the Poverty-of-the-Stimulus argument and that infants and LLMs are computationally distinct learners. Limitations, acknowledged by the authors: no new experiments, a curated paper sample, heterogeneous evaluation pipelines, and no unified performance metric; there are also minor internal inconsistencies (e.g., the GPT-BERT 10M-word BLiMP score is reported as 0.81 in the text but 0.71 in Table 3).
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Models trained on 100 million words or fewer of developmentally realistic data do not consistently reach human-level performance on syntactic benchmarks (best BLiMP: GPT-BERT 0.86 at 100M words vs a ~90% human baseline).
- Most benchmark learning happens below 10M words; scaling from 10M to 100M yields only marginal gains, and no gains on constructs where models already fail.
- Masked language models (BERT variants) systematically outperform causal language models (GPT-2) on minimal-pair benchmarks at comparable data scales.
- The proportion of transcribed speech and child-directed speech in training corpora has no consistent effect on benchmark performance, which the authors take to show infants and LLMs are computationally distinct learners.
- Existing NLP evidence, including all BabyLM results, does not support claims that LLMs refute the Poverty-of-the-Stimulus argument.
Strengths
Novel epistemological angle on an active program
To the authors' knowledge the first systematic philosophy-of-science treatment of BabyLM (§1, §5), cleanly separating the 'proofs-of-concept' and 'reverse-engineering' uses of LLMs and stating precisely what each conception can and cannot conclude — e.g., that neither PoS refutation (Piantadosi 2023) nor UG necessity follows from current results (§5.1).
Concrete corpus-composition audit
Table 1's TS/CDS column is a genuine audit: transcribed speech spans 0% (Wikipedia baselines) to 100% (CHILDES), the Second BabyLM winner trained on only ~19% TS / 10% CDS, and the third-challenge 10M winner had 0% CDS — documenting that 'developmentally plausible' is undefined in practice (§3.1).
Explicitly hedged, well-defended conclusions
Verifiable synthesis via appendix score tables
Weaknesses
No new experiments or unified pipeline
Internal numeric inconsistencies
Thin evidential base for the headline CDS/TS claim
Curated, unquantified paper sample
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the BabyLM challenge?
The BabyLM Challenge, started in 2023, is a community effort to train language models on developmentally realistic corpora — under 10 million words (Strict-Small) or under 100 million words (Strict) — and evaluate them on syntactic benchmarks, in order to model how children acquire syntax from limited input.
Do LLMs disprove Chomsky's poverty-of-the-stimulus argument?
According to this paper, no. Models trained on child-scale data (under 100M words) still underperform humans on syntax benchmarks — best BLiMP score is 0.86 vs a roughly 90% human baseline — and they fail specifically on poverty-of-the-stimulus constructs like question formation. So current models neither refute nor prove the argument.
Does training on child-directed speech (like CHILDES) help models learn syntax better?
The paper finds no consistent effect: for example, GPT-2 trained on 4.3M words of CHILDES and GPT-2 trained on 4.3M words of Wikipedia scored identically (0.61) on both BLiMP and Zorro, and BabyLM winners used corpora with as little as 0-10% child-directed speech without obvious penalty.
Which benchmarks are used to test BabyLM models on syntax?
Mostly minimal-pair benchmarks: BLiMP (67,000 pairs) and Zorro, plus PoSH-BENCH (PoS-specific constructs like wanna-contraction and question formation), MSGS (surface vs linguistic generalization), HierQ (hierarchical yes/no question formation), and PG/ATB wh-movement tests from Lan et al. (2024).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (BLiMP scores vs corpus size) cannot be analyzed from extracted text; only its caption is available, so the MLM-vs-CLM and diminishing-scale visual claims rest on the caption plus Tables 3-6.
- Table 1 extraction is partially garbled (model/data/benchmark column alignment), so per-paper corpus attributions were cross-checked against the prose of Section 3.1.
- The flagged 0.81 vs 0.71 GPT-BERT inconsistency could partly stem from text-extraction noise, though both values appear cleanly in the provided text.
- Appendix A notes the authors themselves could not find full model specifications for Yedetore et al. (2023), limiting one row of the comparison.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.