arXiv paper2026-08-31· machine learning (LLM evaluation & interpretability)· benchmark

Autoregressive Mosaics: Probing 2D Spatial Reasoning in Text-Only Language Models

Q: What is AM-Bench?

AM-Bench is a benchmark that tests whether text-only large language models can reason about 2D spatial layout. Models write short programs that render 24×24 pixel mosaics through a custom six-primitive drawing API. A translation task gives the complete geometry in words (isolating code-expression skill), while a layout task gives an underspecified prompt such as 'the flag of Germany' (requiring the model to compose the arrangement itself), scored by two vision-language judges with human validation.

Q: Can LLMs trained only on text draw images?

According to this paper, yes but unevenly. All eight tested open-weight models (8B–34B) reliably converted fully described geometry into working drawing code, with a perfect median part-IoU of 1.0. Yet their ability to compose an image from an underspecified prompt varied widely, from 3.57 down to 1.75 on a 0–5 judge scale — showing that spatial composition, not code writing, is the bottleneck.

Q: Do LLMs have internal spatial representations, like Othello-GPT's board representation?

Only partially. Before generation, a linear probe can decode a coarse 6×6 occupancy map exceeding a text baseline in all eight models, but it reflects only the layout implied by the prompt. The model-specific layout actually drawn is not decodable (R² ≤ 0.005), and behavioral tests show the model decides the layout incrementally during generation rather than executing a fixed pre-formed plan.

Q: Why does asking an LLM for SVG produce better images than code?

The paper's medium ablation shows raw SVG beats its custom canvas API for every model tested (+0.37 pooled on a 0–5 scale even after matching output resolution), likely because models saw extensive SVG markup during pretraining. The authors argue this data exposure inflates SVG scores, which is why they use a custom, uncontaminated canvas API to measure genuine zero-shot spatial composition rather than memorized patterns.

Ashwin Nedungadi, Stefan Oehmcke, Stefan Lüdtke

View original paper ↗arXiv:2608.30751

cs.AI · cs.CV

Quick verdict

AM-Bench separates translation from 2D layout composition in 8 text-only LLMs; careful work, but VLM-judge noise and 2-model probe scope cap the claims.

Summary

The paper introduces AM-Bench, a benchmark asking whether text-only LLMs hold an internal representation of 2D spatial layout or merely translate verbal descriptions into drawing code. Images are represented as 24×24 'autoregressive mosaics' produced via a custom six-primitive canvas API, chosen to avoid SVG pretraining contamination. A translation task supplies complete geometry in words and is scored by exact-area part-wise IoU (PIoU); a layout task gives underspecified prompts and is scored by two VLM judges (Qwen2.5-VL-7B, InternVL3-8B) validated on 350 human-judged pairs. Across eight open-weight models (8B–34B), all pass translation (median PIoU 1.0, chance 0.072) while layout scores spread widely (J1 overall 3.57 for GLM-4 32B down to 1.75 for CodeLlama 34B); judges agree exactly on model ranking but only moderately per-sample (r=0.43–0.54), and human agreement is moderate (Krippendorff α=0.60). An unexpected T2<T3 inversion suggests 24×24 resolution limits iconic rendering. Replacing canvas code with SVG raises scores (pooled +0.37 resolution-matched, six of eight models significant), showing medium dependence. Linear probes decode a coarse 6×6 occupancy above a TF-IDF baseline in all eight models (mean ΔR²=+0.164), but only the prompt-implied shared component, not the model-specific layout; a truncation-and-displacement behavioral test shows models track the evolving textual geometric state during generation. Limitations: judge noise, two-annotator validation, and decomposition/causal tests covering only two models.

Review complete

Assessment at a glance

machine learning (LLM evaluation & interpretability)

risk flags identified

● high 0

● medium 3

● low 2

Reproducibility readinesshigh

Code & data availableyes

Review confidencehigh

AI text signal0/20 passages flagged

Deep research viableCode-reproducibleNeeds GPU

5 strengths · 5 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

All eight open-weight text-only LLMs reliably express fully specified geometry as canvas code (median PIoU 1.000 vs 0.072 chance), so layout-score differences reflect spatial composition rather than code-generation ability.
The output medium is an active constraint: resolution-matched raw SVG raises layout scores for every model (pooled +0.37, 95% CI [0.26, 0.47]; six of eight individually significant), so absolute canvas scores are a property of the interface.
Pre-generation residual-stream activations encode only the prompt-implied generic layout (shared component R² 0.550/0.573), not the model-specific layout actually drawn (R² ≤ 0.005 against a 0.88–0.94 reliability ceiling).
During generation, models track the evolving textual geometric state rather than executing a fixed plan: coordinate displacement raises continuation rates (GLM-4 32B 43.8%→84.6%; Gemma 2 27B 2.5%→41.2%) with a graded dose-response by displacement size.

Strengths

Clean control-task architecture
The translation task (Sec 3.2, Table 1) fully separates code expression from spatial composition: all eight models reach median PIoU 1.000 (chance 0.072, τ=0.6), licensing the paper's reading of layout variance (J1 3.57 to 1.75) as compositional rather than syntactic.
Confound-closed medium ablation
Exp 2 re-rasterizes all SVGs at 24×24 from vector sources and re-scores the canvas arm under a medium-neutral judge prompt (A.2.4), isolating expressiveness from rendering fidelity; pooled resolution-matched delta +0.37 (95% CI [0.26, 0.47]) vs the confounded native +0.49 (Table 6).
Above-standard probing hygiene
Rigorous behavioral causal test
Transparent failure reporting

Weaknesses

VLM-judge dependence of all layout numbers
Thin human validation
Interpretability claims broader than the evidence base
Translation robustness is pilot-scoped
Roster and prompt-suite constraints

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

AM-Bench (arXiv 2608.30751v2) targets a narrowly posed but fundamental question: do text-only LLMs hold an internal representation of 2D spatial layout, or merely translate verbal spatial descriptions into drawing code? The benchmark decomposes generation into a four-stage pipeline (prompt → internal plan → program code → rasterized 24×24 mosaic) and probes each link: a translation control task (complete geometry given in words; scored symbolically by exact-area part-IoU, τ=0.6), an open-ended layout task (two VLM judges, five 0–5 dimensions, pass ≥3.0), a medium ablation (canvas code vs raw SVG), and activation probing plus a behavioral intervention. Central claims: (1) all eight open-weight models (8B–34B) pass translation (median PIoU 1.000 vs 0.072 chance), so layout differences are compositional; (2) SVG lifts every model's layout score (pooled +0.37 resolution-matched); (3) pre-generation activations encode only the prompt-implied shared layout, not the model-specific one; (4) generation is governed by incremental tracking of the textual geometric state, not a pre-formed plan.

Methodology Assessment

The control-task architecture is the paper's core asset: because translation is passed by every model (Table 1), the 3.57-to-1.75 layout spread is legitimately attributed to composition. The anti-contamination rationale for the custom six-primitive canvas (Sec 3.1) is well-motivated and empirically vindicated by Exp 2. The symbolic PIoU metric (Eq. 1) is resolution-independent, with a measured chance floor (0.072 over 14,500 random placements) and an over-paint penalty. The medium ablation closes both obvious confounds — rendering fidelity (24×24 re-rasterization) an

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 5 strengths & 5 weaknesses, analyzed in detail
5 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is AM-Bench?

AM-Bench is a benchmark that tests whether text-only large language models can reason about 2D spatial layout. Models write short programs that render 24×24 pixel mosaics through a custom six-primitive drawing API. A translation task gives the complete geometry in words (isolating code-expression skill), while a layout task gives an underspecified prompt such as 'the flag of Germany' (requiring the model to compose the arrangement itself), scored by two vision-language judges with human validation.

Can LLMs trained only on text draw images?

According to this paper, yes but unevenly. All eight tested open-weight models (8B–34B) reliably converted fully described geometry into working drawing code, with a perfect median part-IoU of 1.0. Yet their ability to compose an image from an underspecified prompt varied widely, from 3.57 down to 1.75 on a 0–5 judge scale — showing that spatial composition, not code writing, is the bottleneck.

Do LLMs have internal spatial representations, like Othello-GPT's board representation?

Only partially. Before generation, a linear probe can decode a coarse 6×6 occupancy map exceeding a text baseline in all eight models, but it reflects only the layout implied by the prompt. The model-specific layout actually drawn is not decodable (R² ≤ 0.005), and behavioral tests show the model decides the layout incrementally during generation rather than executing a fixed pre-formed plan.

Why does asking an LLM for SVG produce better images than code?

The paper's medium ablation shows raw SVG beats its custom canvas API for every model tested (+0.37 pooled on a 0–5 scale even after matching output resolution), likely because models saw extensive SVG markup during pretraining. The authors argue this data exposure inflates SVG scores, which is why they use a custom, uncontaminated canvas API to measure genuine zero-shot spatial composition rather than memorized patterns.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Figure contents (e.g., Fig 3 per-subcategory heatmaps, Fig 4 bars, Fig 12 UMAP scatter) are accessible only through captions and in-text numbers; visual inspection of the rendered images is not possible from this text extraction.
The project-page hyperlink was removed for anonymity (stated in the paper itself), so the promised code/data release cannot be verified from the provided text.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper