List Counting Failures Are Not One Phenomenon
Iyad Ait Hou, Saad Mankarious, Aya Zirikly, Rebecca Hwa
cs.LG
Quick verdict
List-counting failures split into family-specific modes, not one shared bottleneck; evidence is controlled and honest, but transfer claim rests on one lever.
Summary
The paper asks whether large language models fail at counting items in a bracketed list for one shared reason. Across seven open-weight instruct models (Qwen 2.5 14B/32B/72B, Gemma 2 27B, OLMo 2 32B, Llama 3.1 70B, R1-Distill-Qwen-32B) evaluated on 1,300 identical prompts spanning five surface conditions, wrong answers form family-specific modes: Qwen and Gemma 27B flip odd lengths to nearby even integers, OLMo concentrates errors on a few mid-range integers, and Llama under-counts. Contrary to tokenization-load accounts, lists of random strings (~4x more tokens per item) are more accurate than common words for every model (delta +0.07 to +0.24). Linear probes on the answer-position residual recover the true count at 0.85-0.96 on model-wrong cases versus a ~0.07 permutation baseline, indicating late emission failure. A late-MLP scaling intervention (reverse_half) yields selection-corrected held-out fix rates of 0.20-0.25 on Qwen but only 0.055 on Gemma 27B, while residual steering helps all models only as a zero-sum trade. A held-out test on Mistral-Small-24B missed, undermining concentration-based transfer prediction. Limitations include lengths capped at 15, a single English ask as the main protocol, 4-bit quantization for the 72B-class models, no identified training-data cause, and code promised only upon acceptance.
Review complete
Assessment at a glance
0
risk flags identified
● high 0
● medium 0
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On identical list-cardinality prompts, wrong answers form family-specific modes (Qwen/Gemma-27B odd-to-even flips, OLMo mid-range concentration, Llama under-count) rather than one shared failure pattern.
- Heavier subword fragmentation does not make counting harder: all seven models are more accurate on RANDOM_STRINGS (~4x tokens/item) than on COMMON_WORDS (delta +0.07 to +0.24).
- When the emitted answer is wrong, a linear probe still recovers the true count from the last-layer residual at 0.85-0.96 round-accuracy against a ~0.07 permutation baseline, indicating late emission failure rather than a missing internal count.
- Models sharing the same odd-to-even error need not share a late-MLP magnitude lever: reverse_half yields 0.20-0.25 held-out fix on Qwen but 0.055 on Gemma 27B, while residual steering works on both only as a zero-sum trade-off.
- Error modes strengthen across released post-training stages (Qwen base-to-instruct; OLMo base-DPO-instruct), but no training-document statistic that installs the attractor integers was identified.
Strengths
Genuinely controlled cross-model design
All seven models see the same 1,300 seeded prompts with only the chat template varying (§3.2, Appendices A/K), plus a tool-argument ecological pilot preserving the drop (§4.5) — this cleanly licenses the cross-family error-mode comparison in Table 3.
Unusually honest negative and disconfirming results
Gemma 9B breaking the family law (gap -0.30 vs 27B's +0.25, Appendix B), the rejected n=8 Gemma vignette (~0.69, Appendix G), and the Mistral-Small-24B held-out miss (0.28 vs predicted null, §4.5) are all disclosed rather than buried.
Well-controlled probe methodology
Selection-corrected causal estimates
Self-deflating Table 4 analysis
Weaknesses
Quantization confound inside the core panel
Transfer conclusion from a single intervention family
Narrow main-protocol scope
No multiplicity control across many cell contrasts
Code not yet released
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper studies *list cardinality* — reporting how many whitespace-separated items appear in a bracketed list — as a controlled probe of why open-weight chat models miscount. Seven instruct models (Qwen 2.5 14B/32B/72B, Gemma 2 27B, OLMo 2 32B, Llama 3.1 70B, R1-Distill-Qwen-32B) see 1,300 identical prompts: one fixed English ask, lengths 3-15, five surface conditions (Table 2). Four claims are staked out: (1) errors form family-specific modes rather than one shared failure (Table 3: even-odd gap +0.40 for Qwen-32B vs -0.24 for Llama-70B); (2) RANDOM_STRINGS is *easier* than COMMON_WORDS for every model (delta +0.07 to +0.24), against tokenization-load accounts; (3) the true count stays linearly decodable when the model errs (0.85-0.96 on model-wrong subsets, Table 5); (4) a late-MLP magnitude fix (reverse_half) transfers poorly — Qwen 0.20-0.25 held-out fix rate vs Gemma 27B 0.055 — while class-mean steering helps only zero-sum (§4.4). A fifth strand is honestly inconclusive: post-training stages modulate modes (§4.5), but no installing training signal is found.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 0 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why do LLMs fail to count items in a list?
This paper finds failures are family-specific emission biases, not one shared input bottleneck: on identical prompts Qwen and Gemma 27B flip odd counts to nearby even numbers, OLMo clusters on a few mid-range integers, and Llama under-counts. When the model is wrong, the correct count usually remains linearly readable from the residual stream (probe accuracy 0.85-0.96 vs a ~0.07 permutation baseline).
Is LLM list-counting failure caused by tokenization?
Not on this benchmark. Lists of random strings carrying about 4x more tokens per item are MORE accurate than common-word lists for every one of the seven models tested (delta +0.07 to +0.24) — the opposite of what a tokenization-load account predicts. The authors scope this as a constraint on input-side stories for this task, not a blanket refutation.
Can a counting fix found in one model transfer to another model with the same error?
Not automatically. A late-MLP scaling fix (reverse_half) gives selection-corrected held-out fix rates of 0.20-0.25 on Qwen 32B/14B but only 0.055 on Gemma 27B despite both models sharing the same odd-to-even error. Residual steering can move all affected models, but only as a zero-sum trade (odd-count gains offset by even-count losses), so any cross-model fix needs a transfer check.
Is code or data available for this paper?
Not yet. The paper states that evaluation scripts, probe/intervention drivers, raw result CSVs, and figure code will be released upon acceptance; seeds, layer indices, and full protocols are documented in Appendices A, G, and K, so a rerun is feasible once the release lands.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure images (Figs. 1-12) appear only as captions in the extraction; visual claims (e.g., logit-lens trajectory shapes) were assessed from captions and tables, not the images.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.