Distance generalization in transformers: why bother with positional encoding?
Daniel Henrik Nevermann, Claudius Gros
cs.CL
Quick verdict
NoPE beats RoPE/ALiBi at distance generalization on delay-copy tasks; controlled and honest study, but narrow tasks and best-of-5 selection limit strength.
Summary
The paper studies distance generalization in transformers: keeping the context length fixed (T=256) while evaluating on inter-token delays unseen during training. Using a task-switching framework with a random task and two delay-copy tasks — full copy (C) and selective copy of even tokens (S) — the authors train 8-layer, 512-dimensional decoder-only transformers with RoPE, ALiBi, or NoPE and probe accuracy as a function of source-recall delay. Three findings emerge. First, NoPE outperforms both explicit positional encodings on out-of-distribution delays (Fig. 2), echoing length-generalization results, though Appendix B/Fig. 7 shows NoPE collapses in small models. Second, enlarging the set of training distances (dmax from 5 to 50) raises absolute out-of-distribution accuracy but yields diminishing relative returns, measured by the ratio P of cumulative out-of- to in-distribution accuracy (Eq. 2). Third, transfer between C and S is mixed: mostly negative for RoPE, positive at moderate OOD but negative at large OOD for ALiBi/NoPE (Fig. 4); with well-separated training ranges, transfer is asymmetric — constructive from S to C yet destructive within C's own training region (Fig. 5). Limitations include only two synthetic tasks, a single architecture scale in the main results, teacher-forced evaluation rather than free-form generation, a best-of-five-runs selection policy, and no statistical testing or code release.
Review complete
Assessment at a glance
3
risk flags identified
● high 0
● medium 2
● low 1
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- NoPE outperforms RoPE and ALiBi on out-of-distribution delays at fixed context length (Fig. 2), consistent with length-generalization findings, though NoPE breaks down at small model sizes (Fig. 7).
- Increasing the number of training distances (dmax from 5 to 50) improves absolute out-of-distribution performance but yields diminishing relative returns as measured by the ratio P of Eq. 2 (Fig. 3).
- Transfer between the full and selective copy tasks is mixed: mostly negative for RoPE, positive at moderate OOD but negative at large OOD for ALiBi and NoPE (Fig. 4).
- With well-separated training ranges ([15,25] vs [45,55]), transfer is asymmetric: constructive from S to C at large distances but destructive within C's own training region for NoPE/ALiBi, while RoPE shows weak but non-destructive transfer (Fig. 5).
Strengths
Clean problem isolation
Fixing T=256 and varying only the source-recall delay (Sec. 1, Fig. 1) cleanly separates 'can't handle unseen positions' from 'can't handle unseen token dependencies' — a confound the authors correctly identify in length-generalization studies.
Controlled, quantified experimental design
Three-way RoPE/ALiBi/NoPE comparison at fixed m=10 and Nd (Fig. 2), plus an explicit relative metric P = Pout/Pin (Eq. 2) with a dmax sweep {5,10,20,30,50} (Fig. 3) makes the diversity claim testable rather than anecdotal.
Above-average self-critique
Shortcut-resistant task construction
Nuanced transfer findings
Weaknesses
Best-of-5 selection without variance reporting
Narrow empirical scope vs sweeping framing
Teacher-forced accuracy only
No mechanistic evidence for stated hypotheses
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: Distance generalization in transformers (arXiv 2609.11913)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 3 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is distance generalization and how does it differ from length generalization?
Length generalization tests whether a transformer trained on short sequences still works on longer contexts. Distance generalization keeps the context length fixed and changes only the distance between related tokens (e.g., a token to copy and the position where it must be pasted) between training and testing. This isolates whether a model fails because of unseen positions or because of unseen token dependencies.
Do transformers need positional encoding to generalize over distances?
In this paper's synthetic delay-copy experiments (fixed 256-token context, 8-layer model), removing positional encoding entirely (NoPE) generalized to unseen delays better than RoPE or ALiBi. However, the appendix shows NoPE fails on smaller models, where explicit encodings remain stable — so the answer is scale-dependent, and the title's provocation holds only within this regime.
Does more diverse training data always improve generalization?
Not uniformly: training on a wider set of source-recall distances improved out-of-distribution accuracy in absolute terms, but the gain was sub-linear — measured relative to in-distribution performance via a cumulative ratio, returns diminished as the number of training distances grew from 5 to 50.
Is code available for this paper?
The provided text contains no code or data availability statement. The setup (architecture, hyperparameters, dataset construction in Appendix A, Tables 1-3) is detailed enough for independent reimplementation, but exact replication is complicated by the best-of-five-runs selection policy.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-7 are embedded images; their exact curve values cannot be read from the extracted text, so figure-based judgments rest on the prose descriptions.
- The example DRCS sequence promised after 'here an extract of a sequence drawn from DRCS with Nbase = 10' (Sec. 3) is absent from the extraction (a rendered listing), so the exact input format could not be checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.