FreqBLiMP: Frequency-Controlled Minimal Pairs Reveal Robustness and Fragility of LLMs Under Lexical Rarity
cs.CL · cs.AI
Quick verdict
FreqBLiMP adds Zipf-frequency control to BLiMP; evidence for lemma-specific fragility is solid overall, but frequency control breaks in curated paradigms.
Summary
The paper introduces FreqBLiMP, a frequency-controlled regeneration of all 67 BLiMP minimal-pair paradigms across three Zipf-frequency regimes (Head 3.5-5.5, Tail 2.4-3.2, XTail 1.2-2.2), with 1,000 pairs per paradigm per regime (67,000 per regime). A lexical overlay built from Open English WordNet, Wikidata gender data, and VerbNet frames expands BLiMP's vocabulary while preserving template feature bundles. Evaluating Llama-3.1, Qwen2.5, Gemma-4, and Mistral base and instruct models with LP, Template LP, and Yes/No scoring, the authors find that lower frequency monotonically reduces sentence likelihood (Spearman rho .979-.993) but only modestly reduces aggregate contrastive accuracy (mean -1.5 pp Head to XTail). Declines concentrate in paradigms requiring lemma-specific knowledge (e.g., principle A domain 3, -17.9 pp under Yes/No), while agreement paradigms stay stable or improve. Controls include token-length matching, COCA collocational support quartiles, and a 3-annotator, 720-item human audit showing parallel human/model declines in a 'declining' arm while stable controls hold near 98%. Main limitations: curated paradigms lose in-window frequency coverage (Table A1: 0% XTail coverage for raising predicates), the human audit covers 12 of 67 paradigms with independently sampled conditions, and no per-paradigm significance tests are reported.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Regenerating all 67 BLiMP paradigms under three Zipf regimes yields well-separated, verified frequency profiles while preserving each paradigm's grammatical contrast.
- Decreasing lexical frequency monotonically reduces sentence likelihood (Spearman rho = .979-.993) but only modestly reduces aggregate contrastive accuracy (mean -1.5 pp Head to XTail across seven base models, Tables 4/E2).
- Accuracy losses concentrate in paradigms requiring lemma-specific information (principle A domain 3 -17.9 pp, passive 1 -15.1 pp under Yes/No), while overt-morphosyntax agreement paradigms remain stable or improve (e.g., distractor agr. RC +5.0 pp).
- Rare lexicalization removes misleading high-frequency collocational traps rather than amplifying structural interference (attractor penalty falls from 24.3 pp in Head to 17.1 in XTail, Table E3), and COCA support is a bounded covariate (2.2 pp quartile spread, Table 9).
- Instruction tuning makes raw LP readout less interpretable while making grammaticality judgments more accessible via prompting (G4-31B-I: 58.9% LP vs. 88.3% Yes/No, Table 7).
Strengths
Systematic, verified benchmark construction
All 67 BLiMP paradigms are regenerated under three explicit Zipf windows (Table 1: Head 3.5-5.5, Tail 2.4-3.2, XTail 1.2-2.2) with 1,000 pairs each; realized distributions are well separated (Fig. 1) and lexical diversity far exceeds original BLiMP (15,111 XTail lemmas vs. 2,607; top-20 lemma share <5%, Tables 3/C1).
Three scoring interfaces expose interpretable dissociations
LP, Template LP, and Yes/No readouts (Appendix D.1) separate likelihood from judgment: G4-31B-I scores 58.9% under LP but 88.3% under Yes/No (Table 7), and instruction tuning consistently lowers LP accuracy while raising prompted-judgment accuracy (Sec. 5.5).
Confound controls beyond benchmark-averages
Human validation with a stable control arm and honest reporting
Weaknesses
Frequency control fails in exactly the paradigms where large effects are ranked
No inferential statistics at the paradigm level
Thin, confounded human validation
Small aggregate effect with unstable model ordering
External validity limits
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FreqBLiMP (arXiv 2609.07153) is a benchmark-resource paper that regenerates all 67 BLiMP minimal-pair paradigms under three explicit Zipf-frequency regimes - Head 3.5-5.5, Tail 2.4-3.2, XTail 1.2-2.2 (Table 1) - producing 1,000 pairs per paradigm per regime (67,000 per regime) while keeping each paradigm's acceptability contrast fixed. Three claims carry the paper: (1) lower lexical frequency monotonically lowers sentence likelihood (Spearman rho = .979-.993, Fig. 2; word-normalized LP falls 2.58 points on average) yet costs only ~1.5 accuracy points on aggregate (Tables 4, E2); (2) losses concentrate in paradigms needing lemma-specific information - principle A domain 3 falls -17.9 pp and passive 1 -15.1 pp under Yes/No scoring (Table 5) - while agreement paradigms hold or improve; (3) the pattern reflects an interaction of rarity with lexical licensing, not wholesale syntactic failure.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FreqBLiMP?
A frequency-controlled extension of the BLiMP minimal-pair benchmark: all 67 BLiMP paradigms are regenerated with content words drawn from three Zipf-frequency regimes (Head roughly 3.5-5.5, Tail 2.4-3.2, XTail 1.2-2.2), 1,000 pairs per paradigm per regime, while keeping the same grammatical acceptability contrasts. It tests whether language models' grammatical preferences survive when contrasts are built from rare words.
Do LLMs get worse at grammar when sentences contain rare words?
According to the paper, only modestly on average: aggregate minimal-pair accuracy drops about 1.5 percentage points from the high-frequency to the extreme-tail regime across seven open-weight models, even though sentence likelihood falls sharply. The aggregate hides large per-phenomenon variation - contrasts requiring lemma-specific knowledge (passives, binding, argument structure) degrade by up to roughly 15-18 points, while agreement paradigms stay stable or improve.
Is the FreqBLiMP dataset and code publicly available?
Yes. The paper states that the dataset, its generation pipeline, and evaluation code are available at github.com/TimeTravelerTy/freqblimp-generation, and reports about 320 H100 GPU-hours for the full evaluation suite, with exact prompts and package versions documented in the appendices.
How was FreqBLiMP validated with humans?
Three native-English annotators judged 720 items from 12 paradigms, split into a 'declining' and a 'stable' arm. In declining paradigms, correct selection fell from 97.7% (Head) to 87.7% (XTail) with rising 'cannot decide' rates, while stable controls held near 98%. The authors note this audits stimulus quality rather than providing a controlled estimate of a human frequency effect.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The excerpt contains three '[...sampling gap...]' regions - within Section 3 (after Table 2, around the dataset-diagnostics discussion), within the Section 5.3-to-5.4 transition (part of the human-validation setup), and within the references/Appendix E.2 region; all judgments here rely only on the visible text.
- Figures 1-4, E1, E2, and H1 are available only via captions; underlying per-bin values were not extractable, so figure-based claims rely on the in-text statistics.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.