Thought without systematicity? Evaluating reasoning models on rule induction tasks
cs.CL · cs.AI · cs.LG
Quick verdict
Shows reasoning models often fail isomorphic variants of tasks they can solve; strong framing, but only five proprietary models and no human baseline.
Summary
Schug and Lake adapt four classic cognitive-science rule-induction paradigms—grammar-based instruction learning, symbolic Raven's progressive matrices, program induction over integer sequences, and Boolean category learning—to test whether reasoning models behave systematically. For each task they generate K=5 structurally equivalent variants using isomorphisms (symbol substitution, feature rebinding, constituent permutation, example reordering) and score models with three monotone metrics: whether any variant is solved, the fraction solved, and whether all are solved, using majority votes over N=5 sampled attempts. Across Gemini 3.1 Flash-Lite/Flash/Pro, Claude Opus 4.7, and GPT-5.5, most task families show a large systematicity gap: e.g., Claude Opus 4.7 solves at least one variant in 91.67% of program-induction variation sets but all variants in only 50.00%. Boolean category learning is the exception, with near-perfect systematicity for nearly all models. A variance decomposition and a greedy-decoding re-run (Gemini only) show sampling noise dominates but the gap persists; varying reasoning effort low/medium/high does not consistently help. Limitations include K=5 variations, exclusive reliance on proprietary frontier models, no human comparison despite the cognitive-science framing, and figures whose exact values are hard to extract from the text. Code is released, and the isomorphism generators are fully synthetic and reproducible.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reasoning models that can solve a rule-induction task frequently fail structurally equivalent isomorphic variants of the same task (the 'systematicity gap').
- The gap is not primarily sampling noise: it persists under greedy decoding for Gemini models even though within-task-variation variance dominates the total variance (Table 1, Figure 7).
- Increasing reasoning effort from low to high does not consistently improve systematicity (Figure 6).
- Boolean category learning is the exception where almost all models are perfectly systematic, showing perfect systematicity is achievable in principle (Section 5.1).
- Lack of systematicity undermines test validity: measured ability in one context is a weak predictor of ability in other contexts (Discussion).
Strengths
Well-grounded formalization of a fuzzy concept
Section 3 defines systematicity as invariance of ability under task isomorphisms τ=(π,σ,φ,ψ), with the transformed rule f̃=ψ∘f∘φ⁻¹, and Section 3.5 derives three monotone metrics (any/fraction/all of K variations) visualized as bullet charts — a clean, reusable measurement framework rather than anecdotal probing.
Tasks anchored in established cognitive-science paradigms
The four families (grammar-based instruction learning from Lake & Baroni 2023, symbolic Raven's matrices, integer-sequence program induction from Rule et al. 2024, Boolean concept learning from Feldman 2000/Piantadosi et al. 2016) inherit decades of validation, and Section 4 describes careful support-set construction (e.g., Hamming-graph boundary sampling with an ideal observer to guarantee rule uniqueness in Section 4.4).
Sampling noise is explicitly modeled, not ignored
Honest, self-aware limitations
The Boolean-learning contrast sharpens the interpretation
Weaknesses
Isomorphism does not guarantee equal difficulty — and the paper's own example concedes this
Only proprietary frontier models, no human baseline
Small sample sizes with no uncertainty quantification on the headline metrics
Some null-cell asymmetries in the variance decomposition
Figure-dependent results are unverifiable from the text
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This preprint (arXiv:2609.13948v1) by Schug and Lake asks whether reasoning models exhibit *systematicity* — Fodor & Pylyshyn's principle that grasping a concept entails grasping its close variants. The operationalization is crisp: four rule-induction families from cognitive science (grammar-based instruction learning, symbolic Raven's matrices, integer-sequence program induction, Boolean category learning), each with compositional structure exploited to generate K=5 isomorphic task variants per transformation type (symbol substitution, feature rebinding, constituent permutation/recomposition, example reordering). The headline claim, supported in Section 5.1, is a robust **systematicity gap**: models that solve a task often fail its structural twins. The flagship number is Claude Opus 4.7 on program induction: ≥1 variation solved in 91.67% of variation sets vs. all solved in 50.00%.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the systematicity gap in reasoning models that this paper identifies?
The paper shows that frontier reasoning models (Gemini 3.1 Flash-Lite/Flash/Pro, Claude Opus 4.7, GPT-5.5) often solve at least one variant of a rule-induction task but fail other structurally equivalent variants of the same task. For example, Claude Opus 4.7 could solve at least one variant in 91.67% of program-induction task variation sets but solved all variants in only 50.00% — a 41.67 percentage-point gap between capability and consistent, systematic capability.
How did the authors create 'structurally equivalent' task variants?
They use task isomorphisms: invertible, structure-preserving transformations such as replacing meaningless symbols (dax→lug), rebinding latent features to different attributes, permuting constituents, and reordering examples. Formally a transformation τ=(π,σ,φ,ψ) maps the task so the new hidden rule is f̃=ψ∘f∘φ⁻¹, guaranteeing the underlying problem structure is unchanged. They sample K=5 such variations per task.
Is the failure just randomness from temperature-based sampling?
Mostly no. The variance decomposition in Appendix A shows within-task sampling noise often dominates total variance, but when the authors re-ran the Gemini models with temperature zero, the systematicity gap persisted essentially unchanged (Figure 7). Majority voting over 5 attempts also failed to eliminate the gap, so the authors conclude the inconsistency goes beyond decoding noise. GPT-5.5 and Claude Opus 4.7 could not be tested at zero temperature because their APIs do not support it.
Which tasks did models handle systematically?
Boolean category learning (classifying objects with a simple rule like 'all red objects are wudsy') was the exception: nearly all models solved all tasks and their isomorphic variations. The systematicity gap was largest for symbolic Raven's progressive matrices and program induction over integer sequences, with grammar-based instruction learning showing modest overall performance except for GPT-5.5, which was either fully systematic or fully failed on each task.
Is the code and data available?
Yes. The paper lists a public code repository (github.com/smonsays/systematicity-eval). All task families are procedurally generated and synthetic, so the task generators and evaluation pipeline are fully reproducible; however, re-deriving the headline model numbers requires paid access to the same proprietary model APIs, and model outputs are stochastic.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1, 6, 7, 8, 9 and 10 are bullet/bar charts whose per-cell values cannot be recovered from the extracted text; conclusions about them rest on the prose descriptions.
- The exact number of task variation sets per task family is not stated in the extracted text, so absolute counts behind the reported percentages are unknown.
- Task prompts (Figures 11-14) are included, but per-task example formatting (e.g., object representations in Boolean learning) may be garbled by extraction.
- The paper is a preprint (arXiv:2609.13948v1) and has presumably not completed peer review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.