AGI Maze Prediction Datasets: A Compact Benchmark for Learning World Dynamics with Transformers
cs.LG · cs.AI
Quick verdict
Proposes compact maze-prediction benchmark; a spatial-memory Transformer reaches 100% on some tasks, but single-run results lack seeds and parameter matching.
Summary
The paper introduces the AGI Maze Prediction Datasets, a supervised benchmark derived from procedurally generated, stateful grid worlds, testing whether Transformers learn transferable action-conditioned dynamics rather than memorizing layouts. Three tasks are defined: per-step next-position prediction, fixed-horizon final-position prediction without intermediate states, and autoregressive prediction of textual observations along trajectories; training and validation mazes are source-maze-disjoint and evaluation uses strict greedy exact-match accuracy. From-scratch baselines include a byte-level GPT-style Transformer, an auxiliary latent-memory Transformer with unstructured cells, and a pseudo-video spatial-memory Transformer maintaining a 2D latent frame updated from action history. The byte model solves simple regimes (1.0000 on 3x3-keys per-step) but degrades with board size, horizon, and river mechanics (0.881 on 3x4-rivers h=2); the auxiliary-memory control fits some training regimes perfectly yet generalizes worse (0.886 vs 0.951 on 4x4-keys h=2); the spatial-memory model reaches 1.000 validation accuracy on 3x3-keys h=4 and 4x4-keys h=2, 0.973 on rivers, and 0.997 trajectory-level text-trace accuracy versus 0.869 for the byte baseline. Main limitations: single-run results without seeds or uncertainty, no parameter-matched architecture comparison, teacher-forced text-trace evaluation, and small task scale; the authors frame conclusions as diagnostic, not mechanistic proof.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Source-maze-disjoint splits with greedy exact-match evaluation distinguish learning transferable action-conditioned dynamics from memorizing transitions in familiar layouts.
- Byte-level Transformers from scratch learn simple maze regimes but need rapidly growing data as board size, horizon, and mechanics increase, staying below perfect exact accuracy on complex families (e.g., 0.9557 on 3x4-rivers even with 10,508 mixed examples).
- An auxiliary Transformer with unstructured latent memory cells can fit training data perfectly yet does not improve — and can hurt — held-out accuracy (0.886 vs 0.951 on 4x4-keys h=2).
- A pseudo-video spatial-memory Transformer with a 2D latent workspace, given no intermediate maps, positions, or state labels, reaches 1.000 validation accuracy on 3x3-keys h=4 and 4x4-keys h=2 and 0.973 on 3x4-rivers h=2, versus 0.979/0.951/0.881 for the byte baseline.
- The same architecture lifts 3x3-keys text-trace trajectory accuracy from 0.869 (byte baseline) to 0.997, suggesting structured task-aligned working memory is a more useful inductive bias than added unstructured latent capacity.
Strengths
Clean generalization control
Source-maze-disjoint train/validation splits (Sec. 2.2) plus greedy byte-exact scoring (Sec. 3.1) operationally separate rule transfer from layout memorization, the exact confound most grid-world evaluations leave open.
Informative negative control
The auxiliary latent-memory model (Sec. 3.3) fits the 3x3-keys training set to zero NLL and 1.000 training accuracy yet validates at 0.975 vs 0.979 (byte) and clearly worse on 4x4-keys h=2 (0.886 vs 0.951, Table 3) — isolating spatial inductive bias, not extra capacity, as the differentiator behind Table 4.
Strict readout design
Quantified data-scaling trends
Honest scoping of conclusions
Weaknesses
Single-run results, no uncertainty
No capacity-matched comparison
Teacher-forced text-trace evaluation
Unreported data efficiency of the proposed model
Narrow external validity and selection risk
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Structured Preliminary Review — AGI Maze Prediction Datasets (arXiv 2609.02339)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the AGI Maze Prediction benchmark?
A supervised dataset family and benchmark built from procedurally generated grid-world mazes (3x3 to 4x5 boards with key-collection or river mechanics). Models are trained from scratch to predict an agent's next position, the final position after an action sequence, or the textual observations along a trajectory. Training and validation mazes come from disjoint source mazes, and predictions must match the target byte-for-byte under greedy decoding.
What is the paper's main result?
A 'pseudo-video' spatial-memory Transformer that maintains a 2D latent map initialized from the input maze and updated from the action history reaches 100% exact validation accuracy on two fixed-horizon tasks (3x3-keys with 4 actions, 4x4-keys with 2 actions) where a standard byte-level Transformer reaches 0.979 and 0.951, and raises text-trajectory accuracy on the 3x3-keys task from 0.869 to 0.997. An intermediate model with unstructured memory cells did not help, indicating the spatial structure is what matters.
Is code and data available for this paper?
Yes. The paper links a public GitHub repository (github.com/Necr0x0Der/agimaze-predict) and describes released JSONL dataset shards (about 20,000 per-step examples, 4,000 training plus 1,000 validation examples per sequence task, and 4,000/1,000 text trajectories), with training configurations given in the paper's tables.
Does this paper prove that Transformers cannot do world modeling?
No. The authors explicitly frame the results as diagnostic. Small byte-level Transformers do learn simple transition regimes, and their failures are specific to larger boards, longer horizons, and river mechanics. The paper's positive claim is narrower: adding a two-dimensional, action-updated working memory is a useful inductive bias for these tasks, not that vanilla Transformers cannot learn world dynamics in general.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's ASCII map and the per-step example render are whitespace-garbled by PDF text extraction; no judgment in this review relies on them.
- Model-configuration columns extract as run-together digits (e.g., 512 128 4 0.00); they were interpreted by the tables' stated column order (C, d, L, p).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.