Generating Instance Generators in PDDL Planning
Nicola J. Müller, Naya Rudolph, Katharina Stein, Jörg Hoffmann, Ayal Taitler, Timo P. Gros
cs.AI
Quick verdict
I2G: LLM-generated PDDL instance generators with test-verified soundness; efficiency strongly beats ASP/RL baselines but its solvability check is incomplete.
Summary
The paper introduces I2G, a pipeline that uses an LLM to write Python instance generators for PDDL planning domains. Motivated by PDDL's lack of any construct for specifying the intended instance set (legality and subset constraints), the authors prompt an LLM with the domain file, hand-written Python legality tests, and first-order-logic subset constraints, then run a test-and-debug loop: candidate instances are screened for code, file, quality, and constraint bugs (subset checks via Z3; solvability filtered by finite hFF), and bug plus diversity reports feed back to the model. On ten IPC 2023 learning-track domains, all selected generators achieved 100% soundness (Table 2) within ~10 minutes per domain at ~$4 total API cost. Recreating the IPC test sets, I2G finished easy/medium/hard tiers in 21 s/2 m/1 h versus 7 h/43 h/no completion for the ASP approach Gr25 (Table 3; with subset constraints 26 s/16 m/9 h, Table 4), and produced 100 instances in ~6 s versus 26–221 s for the RL approach NeSIG, whose policies take 2–6 days to train (Table 5). A case study uses I2G to expose generalization gaps in a GNN policy and in LLM plan generators on restricted subsets. Limitations: finite hFF cannot prove solvability; diversity is a coarse feature-space proxy; the human effort behind legality tests is unquantified; no code release or LLM ablation is reported.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PDDL provides no means to specify legality or subset constraints on the intended instance set, which has made instance generation historically ad-hoc.
- I2G prompts an LLM with a PDDL domain, Python legality tests, and FOL subset constraints, and iteratively produces Python generators whose outputs pass prescribed legality, subset, quality, and efficiency tests at deployment time.
- On ten IPC 2023 learning-track domains, I2G computes generators with 100% sound instance return within 10 minutes and ~$4 total API cost.
- I2G generators scale far beyond Gr25 (ASP) and NeSIG (RL): 21 s/2 m/1 h versus 7 h/43 h/none on IPC test-set recreation, and ~6 s versus 26–221 s for 100 instances.
- I2G enables, for the first time, efficient evaluation of generalized planning methods on arbitrary instance subsets, revealing concrete failure modes of a GNN policy, GenPlan, and GPT-5.4 plan generators.
Strengths
Clear problem formulation filling a real methodological gap
Section 1 cleanly separates legality constraints (fixed per domain) from subset constraints (purpose-dependent), a distinction the paper argues the literature has never made explicit; the motivation (ad-hoc, opaque hand-written generators cited by Helmert 2003, Hoffmann 2005) is convincing.
Soundness enforced by a well-designed test suite, not trusted LLM output
Table 1 defines eleven tests across four bug categories (code, instance file, quality, constraint); subset constraints are checked by translating instances into Z3 with Enum sorts and uninterpreted Boolean functions, and unsatisfiability cores are fed back to the LLM as targeted error messages (Sec 2.2) — a principled use of SMT for the debug loop and a hard gate at deployment.
Orders-of-magnitude efficiency gains at trivial generation cost
Unusually honest failure and limitation reporting
Case study demonstrates practical payoff beyond benchmarks
Weaknesses
The headline 'soundness guarantee' overstates what is checked
Legality tests may smuggle generator conventions into the Gr25 comparison
Diversity metric is a shallow proxy with no uncertainty quantification
No ablations, single LLM, one run per domain
Reproducibility and human-effort accounting gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a genuine methodological gap: PDDL specifies domains (predicates, action schemas) but not the intended instance set. The authors split this into *legality constraints* (fixed per domain, e.g., a block not on itself) and *subset constraints* (purpose-dependent, e.g., a single goal tower), a conceptual distinction they argue the literature has never made explicit. Central claims: (1) I2G generates domain-specific Python instance generators via an LLM test-feedback loop with deployment-time checks; (2) on the ten IPC 2023 learning-track domains, generators emerge within 10 minutes at ~$4 total API cost and return 100% sound instances (Table 2); (3) the resulting generators scale orders of magnitude better than the ASP approach Gr25 and the RL approach NeSIG (Tables 3–5); (4) the method enables evaluating generalized planners on arbitrary subsets (Sec 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is I2G in AI planning?
I2G (Instance Generator Generation) is a pipeline from this paper that prompts a large language model to write a Python program that generates planning problem instances for a given PDDL domain. The program is iteratively tested and debugged, and at deployment an instance is only returned if it passes legality tests, subset-constraint checks (verified with the Z3 theorem prover), quality tests (solvable-looking, non-trivial), and efficiency limits.
Why is automatic instance generation important for PDDL planning?
PDDL defines a domain (predicates and action schemas) but not which instances are legal or interesting, so researchers have always hand-written per-domain generators whose exact constraints are implicit in code. Automatic, constraint-explicit generation supports fair benchmarking, testing planners on larger instances, and evaluating how well generalized planners handle specific instance subsets.
How does I2G compare to previous instance-generation methods like Gr25 and NeSIG?
Gr25 (Grundke, Helmert, Röger 2025) uses answer set programming and scales exponentially with instance size — in the paper's comparison it needed 7 h/43 h for easy/medium IPC test-set recreation and produced no hard instances, while I2G needed 21 s/2 m/1 h. NeSIG (Núñez-Molina et al. 2024/2025) trains reinforcement-learning policies over 2–6 days per domain subset and takes 26–221 s to produce 100 instances, versus ~6 s for I2G — though NeSIG can target instance difficulty without extra input, which I2G cannot.
Are instances produced by I2G guaranteed to be solvable?
Not provably. They are guaranteed to satisfy the legality tests and subset constraints, be non-trivial, and have a finite hFF (relaxed-plan) heuristic value, but finite hFF does not strictly prove solvability once delete effects are considered — a limitation the authors acknowledge in their own footnote 1.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 are vector plots; only axis labels and captions survive extraction, so case-study coverage values rely on the in-text descriptions (numbers in Tables 1–5 are intact).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.