MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents
Vernon Toh, Navonil Majumder, Zhengyuan Liu, Nancy F. Chen, Soujanya Poria
cs.AI · cs.CV
Quick verdict
Benchmark recasts MNIST as glimpse-based partially observable search; findings are coherent and diagnostic, but n=100 cells lack variance estimates.
Summary
The authors introduce MNIST-PRO, a benchmark that turns MNIST digit recognition into a partially observable, glimpse-based search task formalized as a POMDP: agents move a 64x64 window in 32-pixel steps over binarized 224x224 images under 36-step (single digit) or 78-step (two-digit sequence) budgets, deciding where to look, what to remember, and when to commit. Ten multimodal models are evaluated under raw visual history (H=infinity), textual states and metric grid maps (H=1), and programmatically consolidated visual canvases, plus autonomous harness runs with persistent memory. Control accuracy of 83-99% collapses under partial observability: the best native result is Gemini-3.7-Flash at 75.0%/47.0% (Levels 1/2), no model exceeds 30% on Level 2 without a harness, and the harness lifts Gemini-3.7-Flash to 88%/63%. Offline canvas replays and predictor-swap experiments localize failures in state construction and interpretation rather than evidence acquisition; premature stopping and failure to revise early beliefs are pervasive. Strengths are the clean environment design and controlled decomposition experiments; the main limitations are 100-episode cells with no confidence intervals or significance tests, reliance on drifting preview versions of proprietary models with undisclosed sampling parameters, and a self-acknowledged system-prompt confound in harness comparisons.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Near-ceiling MNIST recognition (83.0-99.0% control accuracy) collapses under glimpse-based partial observability: the best native result is Gemini-3.7-Flash at 75.0%/47.0%, and no model exceeds 30% on two-digit tasks without a harness.
- The memory representation carrying the perceptual state (visual history vs textual state vs metric grid map vs visual canvas) materially changes accuracy, with no representation dominant across models.
- Failures stem mainly from perceptual-state construction and interpretation rather than evidence acquisition: offline canvas replays lift frozen trajectories (e.g., Claude-5-Fable 28.0% to 75.0% average) and predictor swaps improve accuracy on identical canvases (66.5% to 84.3%).
- Agents systematically stop early (323 audited Level 2 predictions cover only one digit) and rarely revise incorrect early beliefs (15 of 17 audited cases persist into the final answer).
Strengths
Clean capability isolation
The 2D masked-canvas design with near-ceiling control accuracy (83.0-99.0%, Tables 1-2) and known action semantics removes the control/rendering confounds of embodied benchmarks (Sections 1-2); the budget arithmetic (Eqs. 1-3 yielding 36/78 steps) checks out exactly against the window/step geometry.
Controlled decomposition methodology
Frozen-trajectory offline canvas replays (Table 3: Claude-5-Fable 28.0% to 75.0% average) and the predictor-swap experiment (Section 4: 66.5% to 84.3% on Level 1 when Gemini-3.7-Flash reads GPT-5.6-Sol (xhigh) canvases) cleanly separate evidence acquisition, state construction, and interpretation.
Systematic representation and horizon ablations
Honest negative results and self-critique
Fine-grained failure quantification
Weaknesses
No statistical inference
Imperfect control ceiling
Reproducibility of headline numbers
Unexplained aggregation and coverage gaps
Narrow scaling claims
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MNIST-PRO (arXiv 2608.31022; DeCLaRe Lab NTU + A*STAR) converts MNIST into a partially observable glimpse-search world formalized as a POMDP (Section 2.1): a 64x64 window moves in 32-pixel steps over a binarized 224x224 canvas (224x448 for Level 2's two digits), with budgets of 36/78 steps derived from Eqs. (1)-(3). The paper claims that (1) near-ceiling recognition (83.0-99.0% control accuracy, Tables 1-2) does not transfer to sequential partial observability; (2) the memory representation carrying the perceptual state (raw visual history H=infinity, textual state H=1, metric grid map H=1, consolidated visual canvas) materially changes outcomes with no single winner; (3) failures localize in state construction and interpretation rather than evidence acquisition; (4) agents stop prematurely and rarely revise early beliefs. Claims are appropriately scoped to ten off-the-shelf multimodal models under prompting (Section 3.1), not to trained perception policies.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MNIST-PRO?
MNIST-PRO is a benchmark that turns MNIST digit recognition into a partially observable game for AI agents: the agent sees only a 64x64 pixel window of a binarized 224x224 image, moves it with up/down/left/right steps, and must decide where to look, what to remember, and when to answer. Level 1 is a single digit (36-step budget); Level 2 is a two-digit sequence whose order must be preserved (78-step budget).
What are the paper's main findings?
Models that recognize digits almost perfectly when the full image is shown (83-99% control accuracy) drop sharply when they must explore: the best native result is Gemini-3.7-Flash at 75% (single digit) and 47% (two digits), and no model exceeds 30% on the two-digit task without a tool harness (with a harness, Gemini-3.7-Flash reaches 88%/63%). The main failure causes are premature stopping, poor construction and interpretation of the internal perceptual state, and failure to revise wrong early beliefs.
Is code or data available for MNIST-PRO?
Yes. MNIST is public, the preprocessing and environment parameters are fully specified in the paper, and the authors provide a code repository (github.com/declare-lab/MNIST-PRO). Note that exact numbers for proprietary models depend on live API versions and default sampling settings.
Which memory representation worked best?
None dominated. Full visual history was best for single digits for seven of eight proprietary models, but textual states and metric grid maps did better on two-digit tasks (e.g., Gemini-3.7-Flash grid map 28% vs 22% visual history). The largest gains came from offline canvas consolidation, where the agent's collected glimpses are programmatically stitched into one image after exploration (Claude-5-Fable average 28% to 75%).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures were extracted as linearized text streams, so the panel-to-series mapping inside Figures 3 and 4 is inferred from axis labels and captions rather than verified against rendered positions.
- Mathematical glyphs (script symbols for sets and the lookback variable) were dropped by PDF extraction; they are rendered here as H, S, C, etc., inferred from surrounding context.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.