Reasoning with Image Generation
Nishad Singhi, Hector Garcia Rodriguez, Aditya Arora, Marcus Rohrbach, Anna Rohrbach
cs.CV
Quick verdict
ReImaGin: a training-free agent reasons by generating images, consistently beating text-only and Sketchpad baselines on six visual tasks, though small eval sets temper the claims.
Summary
The paper introduces ReImaGin, a training-free multimodal agent that interleaves textual chain-of-thought with calls to an instruction-tuned image generator. A single free-form generate_image tool, plus programmatic utilities, lets the MLLM create intermediate visualizations (depth maps, occlusion removal, top-down floorplans, trajectory overlays) within a ReAct loop. Across six tasks (BLINK depth, MIRA puzzle, CAPTURE occlusion counting, Spatial457 collision, MMSI multi-view spatial, and a self-introduced path-tracing task), ReImaGin outperforms No Tools and Visual Sketchpad with Gemini-3.1-Pro, GPT-5, and Qwen-3.5-27B; e.g., path tracing 89.0 vs 74.7 (Gemini) and occlusion-counting sMAPE 7.4 vs 12.5 (GPT-5), with Sketchpad sometimes degrading performance. An ablation replacing the generator with a no-op degrades all six tasks, attributing gains to generation itself. On MMSI, best-of-10 image selection lifts accuracy from 51.0 to 59.0 (Gemini). An automatic prompt-optimization loop recovers most handcrafted-strategy gains and transfers across MLLMs. A manual audit of 120 images finds 75% faithful generations, with downstream correctness of 87% vs 43% conditioned on faithfulness. Main limitations: small evaluation sets (MIRA n=26), the headline gain rests partly on a self-introduced benchmark, no significance tests (3-seed standard errors only, some ±0.0 entries), ~4x inference cost ($0.16 vs $0.04/instance), and open-weights generators lagging on spatially precise tasks.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A training-free MLLM agent with one instruction-following image generator as a general visual tool consistently outperforms text-only reasoning and the specialist-tool baseline Visual Sketchpad across six visual reasoning tasks and three MLLMs.
- The gains derive from generative capability itself: replacing generate_image with a no-op under identical prompts and scaffold degrades performance on all six tasks (Appendix I).
- Test-time scaling that spends extra compute on selecting among N=10 generated images improves multi-view spatial accuracy (51.0→59.0 Gemini; 42.0→54.3 Qwen) without altering the reasoning process.
- Effective visual reasoning strategies can be discovered automatically via iterative prompt optimization, recovering most handcrafted-strategy gains and transferring unchanged across MLLMs (Tables 3-4).
- Generation faithfulness is the main bottleneck: answers are correct 87% of the time when the generated image is faithful versus 43% when it is not (120-image audit, Appendix J).
Strengths
Clean causal ablation isolating the generative tool
Appendix I (Table 7) replaces generate_image with a no-op returning the input image while keeping prompts, tool definitions, and scaffold identical; performance drops on all six tasks (e.g., tracing 89.0→77.5, puzzle 42.3→36.5), directly attributing gains to generation rather than scaffold differences that confound the Table 1 baseline comparison.
Breadth of evaluation across models and generators
Three MLLMs (Gemini-3.1-Pro, GPT-5, Qwen-3.5-27B) × three image generators (Nano-Banana-Pro, FLUX.2 [dev], Qwen-Image-Edit-2511) across six tasks (Tables 1, 2, 6), including a fully open-weights pipeline (Appendix H) with per-generator failure modes identified (FLUX.2 depth 89.1 vs 91.7 no-tools; Qwen-Image tracing 76.0 vs 89.0).
Transparent failure analysis with quantified faithfulness
Automatic strategy discovery is well-documented and transfers
Matched reasoning effort against Sketchpad
Weaknesses
Very small evaluation sets undermine per-task claims
Headline gain rests on a self-introduced benchmark
Statistical reporting is thin and internally inconsistent
Gains depend heavily on handcrafted strategy prompts
Cost and practicality asymmetry is downplayed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ReImaGin and how does it work?
ReImaGin is a training-free framework in which a multimodal LLM (e.g., Gemini, GPT-5) solves visual reasoning problems by calling an instruction-following image generation model through a single 'generate_image' tool to create intermediate visualizations — such as depth maps, occlusion removal, top-down maps, or trajectory lines — and then reasons over those generated images, interleaving text and visuals in a ReAct-style loop.
Does ReImaGin require any training or fine-tuning?
No. Both the reasoning MLLM and the image generator are used off the shelf; the visual reasoning strategy is conveyed purely through prompts, which can also be discovered automatically via an iterative prompt-optimization loop costing about $214 per task.
How much better is ReImaGin than existing baselines?
Across six visual reasoning tasks, it beats no-tool reasoning and the specialist-tool baseline Visual Sketchpad by up to 25 points (path tracing: 89.0 vs 74.7 with Gemini) and reduces occlusion-counting error by roughly 40% relative (sMAPE 12.5 to 7.4 with GPT-5). The one exception is depth estimation with Gemini, where Sketchpad's dedicated depth tool still wins (99.2 vs 94.6).
What are the main limitations of this approach?
Generated images are faithful to the requested transformation only about 75% of the time, and answer accuracy drops from 87% to 43% when they are not; inference costs about four times the Sketchpad baseline ($0.16 vs $0.04 per instance); open-weights generators lag on spatially precise tasks; and some evaluation sets are small (e.g., 26 instances), so per-task differences carry statistical uncertainty.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (generated-image examples, Fig. 1 bar chart) are only partially captured by text extraction and were not visually analyzed.
- A '§ Code' footnote marker is present but the actual code URL is not visible in the extracted text, so code availability is inferred from the marker only.
- The very end of the extracted text (Appendix N.2 selector prompt) is truncated.
- The faithfulness audit (75% faithful; 87% vs 43%) relies on author-performed manual labels that cannot be independently verified from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.