Coding Agents for Generalized Task and Motion Planning Problems
Matteo Merler, Bowen Li, Josh Roy, Yichao Liang, Qianwei Wang, Yixuan Huang, Tom Silver
cs.RO · cs.AI
Quick verdict
Claims coding agents beat hand-engineered planners at generalized TAMP across 28 envs; large, well-controlled study with training-data contamination risk.
Summary
The paper presents AgenticGenPlan, a large-scale empirical study of whether off-the-shelf coding agents can solve generalized task and motion planning (TAMP). Claude Code (Opus 5) and Codex (GPT-5.6 Sol, GPT-6 Astra) receive only a task description and sandboxed simulator access, interactively probe the environment within a fixed $20 synthesis budget, and return a program that is frozen and evaluated on 100 held-out instances: 28 environments (25 KinDER, 3 PDDLStream), seven synthesis methods, five runs each — 98,000 evaluation episodes. On the 16 environments with a hand-engineered planner, Astra averages 95% success, Opus 82%, and Sol 56%, versus 47% for the planners; all three also beat one-shot generation and a re-implemented LLMGenPlan (28%). Synthesized programs are an order of magnitude cheaper per instance than planners (2.1 s / 0.5 s versus 29 s). Qualitative analysis shows agents calibrating kinematic models, testing edge cases, and discovering previously unpublished non-prehensile strategies. Giving environment source code raises means (Opus 74%→84%, Astra 86%→95%). Main limitations: fully observed states with no perception; public benchmark code means training-data contamination cannot be excluded; several Dynamic3D environments remain largely unsolved; statistics are limited to min–max over five runs.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Off-the-shelf coding agents synthesize generalized TAMP programs that outperform hand-engineered planners in mean success on the 16 planner-available environments (56–95% vs 47%).
- Agentic interaction — probing, custom tests, revision — is the decisive ingredient: with identical source code and budget, LLMGenPlan reaches 28% versus 84% for Opus + source.
- Frozen synthesized programs are an order of magnitude cheaper per instance than planners (2.1 s / 0.5 s vs 29 s) while degrading more gracefully as object counts grow.
- Agents discover manipulation strategies absent from any published solution on these benchmarks, providing qualitative evidence against pure training-data recall.
Strengths
Unusually large and systematic evidence base
28 environments (25 KinDER + 3 PDDLStream), 7 synthesis methods, 5 independent runs each, 100 held-out instances per program — 98,000 episodes (Sec. I, IV-A). Evaluation seeds are randomly generated and verified unused during synthesis; sandbox isolation is red-teamed (Sec. III-A).
Clean frozen-program protocol isolating synthesis from test time
The program is frozen and no LLM is invoked at evaluation (Sec. II-C), removing test-time LLM variance; the $20 matched budget bounds synthesis cost comparably across methods.
Well-designed ablations separating information from interactivity
Rich, traceable qualitative evidence
Efficiency and failure reporting, not just success
Weaknesses
Training-data contamination cannot be excluded
No inferential statistics
Potentially weakened key baseline
Subset-inconsistent headline comparisons and outcome-conditioned efficiency
Benchmark provenance overlap
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper reports **AgenticGenPlan**, a systematic empirical study (not a new algorithm) asking whether off-the-shelf coding agents can act as *generalized* TAMP solvers. Configurations: Claude Code with Opus 5, and Codex with GPT-5.6 Sol / GPT-6 Astra, each in a **main setting** (task description + sandboxed simulator access only) and a **+ source** reference setting (environment code visible). Central claims: (1) all three agent configurations beat hand-engineered planners in mean success on the 16 environments with a planner (56–95% vs 47%); (2) they beat one-shot generation and a re-implemented LLMGenPlan across all 28 environments; (3) frozen synthesized programs are an order of magnitude cheaper per instance than planners (2.1 s / 0.5 s vs 29 s) and scale better with object count; (4) agentic interaction — probing, custom tests, revision — is the key ingredient; (5) agents discover previously unpublished manipulation strategies (Fig. 2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can LLM coding agents solve task and motion planning (TAMP) problems?
In this study, yes in simulation: programs written by Claude Code and Codex agents, then frozen with no LLM at test time, achieved 56–95% mean success versus 47% for hand-engineered TAMP planners on the 16 environments where a planner exists, and beat LLM-based baselines across all 28 environments. Dynamic 3D tasks remained largely unsolved by some agents.
What is generalized TAMP?
Generalized task and motion planning aims to produce one reusable solution — for example, a program — that works across many problem instances of the same task family (varying object counts and configurations), instead of re-planning each new instance from scratch with a bespoke planner and hand-designed models.
Is the code and data for this paper available?
The authors state they release all code, including the full prompts given to the agents, and the study builds on public benchmarks (KinDER and PDDLStream). Synthesis used commercial models (Claude Opus 5, GPT-5.6 Sol, GPT-6 Astra) at a stated $20 budget per run, so exact replication depends on those APIs.
Does the LLM run at test time in this approach?
No. The coding agent is used only during synthesis, within a fixed budget; the resulting program is then frozen and evaluated on unseen instances without any LLM calls, keeping evaluation fast — about 0.5–2.1 seconds per instance versus 29 seconds for the planners.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure content (Figs. 1–4) is not analyzable from the extracted text; only captions were available, so visual claims about trajectories and plots rest on captions and prose.
- The abstract in the supplied metadata is truncated mid-sentence; judgments are based on the full-text extraction, which appears to include all sections, tables, and references.
- Min–max rows in Tables I and II may suffer minor extraction misalignment between value rows and interval rows.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.