Type Diversity Enables Transformers to Generalise Compositionally
Anssi Moisio, Mathias Creutz, Mikko Kurimo
cs.CL
Quick verdict
Claims low type diversity, not Transformer limits, explains poor structural generalisation in COGS; evidence is convincing despite high seed variance.
Summary
The paper challenges the finding that structural generalisation is harder than lexical generalisation for seq2seq Transformers. The authors hypothesise the gap stems from dataset design: lexical types (nouns, verbs) have high type diversity — many constructors — while structural types (e.g., noun phrases) have few. Using Grammatical Framework they generate linguistically diversified variants of COGS, SLOG, and over 100 SCAN variants. In the COGS PP-in-Obj→PP-in-Subj task, incrementally adding NP constructors (participial phrases, adjectives, relative clauses, plurals, artificial NPs) lifts accuracy from ~2% to 92.4% across 12 seeds. In SCAN, with lexical (V), phrasal (VP) and clausal (Imp) diversities matched, accuracy correlates with type diversity equally for lexical and structural test cases (0–3% at low diversity, 60–90% at high). They also show atom and compound divergences (DBCA) correlate positively with accuracy in their setups — contradicting Keysers et al.'s claim that compound divergence captures task difficulty — and demonstrate that surface properties such as semantic term order still matter in SLOG. Limitations: very high seed variance (up to ±18.2), a single hyperparameter configuration, ~1M-parameter models only, and a type-diversity metric that depends on grammar design choices.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The reported hardness of structural generalisation relative to lexical generalisation in COGS-type benchmarks is explained by low type diversity (few constructors) of the tested structural types, not by inherent Transformer limitations.
- When lexical and structural type diversity are matched in SCAN, accuracy correlates with type diversity equally for lexical and structural test cases (0–3% at low diversity vs 60–90% at high).
- In datasets where divergence arises from added training diversity, atom and compound divergences correlate positively with accuracy, contradicting Keysers et al.'s proposal that compound divergence captures core CG difficulty.
- Sub- and supertree type diversity helps generalisation when the target type's own diversity is low, and can hurt it at high target diversity under fixed dataset size.
- Surface properties of the logical form, such as semantic term order, independently affect results (duplicated '?' errors drop from 63.2% to 18.1% under GF order in indirect-object wh-questions).
Strengths
Linguistically principled, computable construct
Type diversity is defined as constructor counts in a GF abstract syntax (Sec. 5.2), giving a precise metric; the paper candidly discusses grammar-relativity and the CN/Det/Num→NP collapse rather than hiding it.
Diagnostic preliminary experiment
Table 1 (Sec. 5.1): PAPs restricted to subject with acl semantics collapse accuracy to 0.2 (±0.4) while the nmod variant is position-insensitive (83.8 vs 81.8), cleanly separating WMP's surface-order account from a constructor-diversity account.
Systematic confound control
Breadth and scale of evidence
Reproducibility posture
Weaknesses
Statistical fragility of the trajectory
TD never fully isolated from surface correlates
Grammar-relative metric without sensitivity analysis
Limited external validity
Heterogeneous, sometimes marginal SLOG gains
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.13144) targets an influential claim in the compositional generalisation literature: structural generalisation (recombining multi-word structures into new roles) is intrinsically harder for seq2seq Transformers than lexical generalisation (Kim & Linzen 2020; Yao & Koller 2022; Li et al. 2023; Jabbar et al. 2025). The counter-hypothesis (Section 5): the gap is a dataset artefact — lexical types have hundreds of constructors while the tested structural types (NP with PP, in the COGS PP-in-Obj→PP-in-Subj task) have one or two. 'Type diversity' (TD) is operationalised as constructor counts in a Grammatical Framework grammar and tested on regenerated COGS/SLOG variants plus 120 SCAN variants.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'type diversity' in this paper?
It is the number of distinct constructor functions in a Grammatical Framework grammar that build expressions of a given type (e.g., how many different ways a noun phrase can be constructed: simple, with a prepositional phrase, with a relative clause, etc.). It measures structural variety independent of which specific words populate the phrases.
Does this paper show Transformers can do compositional generalisation after all?
Partially: it shows that the notoriously poor structural generalisation on benchmarks like COGS largely tracks the scarcity of structural constructors in the training data; when type diversity is matched in SCAN, lexical and structural generalisation perform similarly. Accuracy still plateaus below 100%, so the paper does not claim human-level compositional ability.
Which datasets were used and is the data available?
Grammatical Framework-regenerated variants of COGS, SLOG, and over 100 variants of SCAN. The data and code are publicly available at github.com/anmoisio/type-diversity.
Why does the paper contradict the CFQ compound-divergence finding?
In CFQ (Keysers et al., 2020), higher compound divergence meant the test set required more unseen combinations, correlating with worse accuracy. Here compound divergence rises because the training set gains more constructors while the test set stays fixed, so divergence signals broader training support rather than a harder test — the authors argue the metric conflates these two situations.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with three '[...sampling gap...]' markers; parts of Section 6, the tail of Section 8.1.3, part of Section 9.3/10, and portions of the references were not visible — all findings cited above come from the visible sections only.
- Figures 1–7 are present only as captions in the extracted text; plotted values beyond the correlation coefficients quoted in captions could not be inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.