Grounded Evaluation and Repair for NL-to-PDDL Problem Generation
Joana Rosa, Pedro Santos, Valdemar Oliveira, Romão Silva, L. Miguel Silveira, Bruno Martins
cs.AI
Quick verdict
Solid empirical study: LLM NL-to-PDDL operational success diverges from benchmark reconstruction; honest analysis but small N, no significance tests.
Summary
The paper presents an empirical study of an end-to-end pipeline that translates natural language planning descriptions into PDDL problem instances with LLMs (mainly gpt-4.1-mini; gpt-5.5 and opus-4.7 in appendices). Each candidate problem passes through parsing, a domain-conformance checker, planner execution (Fast Downward, TFD, or ENHSP) with VAL plan validation, and an LLM critic, followed by up to two repair iterations. The central methodological move is separating online 'operational success' from offline benchmark-reference reconstruction, measured by renaming-invariant structural matching and, where supported, Planetarium semantic equivalence. Across Planetarium, AutoPlanBench, and six curated PDDL 2.1 domains, the two signals diverge sharply: few-shot PDDL 2.1 reaches 0.733 operational success but only 0.067 structural match. Few-shot prompting helps Planetarium and PDDL 2.1 but hurts supported AutoPlanBench (semantic/structural 0.842 to 0.714), and repair lowers structural matching on unsupported APB (gains -0.100 and -0.067). A 24-case manual audit finds the critic agrees with human faithfulness judgments in only 13/24 cases. Key limitations: 60-instance samples without significance tests, no comparison against prior NL-to-PDDL systems, template-generated APB descriptions, and reliance on a given correct domain file.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Parseability and planner success substantially overestimate faithfulness: operational acceptance and benchmark-reference reconstruction are distinct, divergent signals across three benchmark families.
- A unified generate-check-repair loop (parsing, domain conformance, planning, VAL validation, LLM critic) improves operational success, with the largest gains on Planetarium and PDDL 2.1.
- Few-shot prompting is beneficial but not uniform: it helps Planetarium, unsupported APB, and PDDL 2.1, while degrading supported APB (final semantic/structural 0.842 to 0.714).
- PDDL 2.1 reference reconstruction remains very hard: structural match is at most 0.067 with gpt-4.1-mini even at 0.733 operational success, and at most 0.250 with gpt-5.5/opus-4.7.
- The LLM critic is a useful operational feedback signal but not a reliable faithfulness oracle, agreeing with manual judgment in only 13/24 audited cases.
Strengths
Evaluation-signal decoupling is the real contribution
Section 3.2/4.4 cleanly operationalize the online/offline split with divergence rows (Op. Non-Struct., Struct. Non-Op.), turning the vague faithfulness worry into measurable gaps, e.g., 0.500 operational vs 0.000 structural on baseline PDDL 2.1 (Table 3).
Cross-formalism coverage with planner-appropriate backends
The same generate-check-repair loop drives classical domains (Fast Downward) and PDDL 2.1 (TFD for time-simple, ENHSP for numeric) across three benchmark families, with size statistics for every set given in Section 4.1.
Unusual transparency about the critic
Replication materials are unusually complete
Honest negative results on few-shot prompting and repair
Weaknesses
No inferential statistics on small samples
No external baselines
Critic-generator same-model confound with a weak audit
Unexplained structurally-matched-but-not-operational cases
Benchmark circularity and confounded few-shot selection
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks how NL-to-PDDL *problem* generation (given a correct domain.pddl) should be evaluated and repaired. Its three stated questions (Section 1) are: deployment-realistic evaluation with few-shot prompting and iterative repair; the size of the gap between operational acceptance and reference-based reconstruction; and transfer to PDDL 2.1 with temporal/numeric constructs. The central empirical claims are that parse/planner success overestimates faithfulness (operational success and reference reconstruction diverge, e.g., 0.500 vs 0.000 on baseline PDDL 2.1, Table 3); that a five-check acceptance criterion (parse, domain conformance, solve, VAL, LLM critic) plus repair improves operational success; that few-shot prompting is non-uniform (helps Planetarium and PDDL 2.1, hurts supported APB: final semantic/structural 0.842 to 0.714, Table 1); and that the LLM critic is useful but not a faithfulness oracle (13/24 audit agreement, Section 5.5). These claims are clearly scoped and mostly supported by the reported numbers — though only descriptively.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'operational success' mean in this paper?
A generated PDDL problem is accepted only if it parses, passes a static domain-conformance check, a planner (Fast Downward, TFD, or ENHSP) finds a plan, VAL validates that plan, and an LLM critic judges it faithful to the natural language description. No reference solution file is needed, so it can run online inside the repair loop.
Why do operational success and benchmark matching diverge so much?
A problem can be parseable and solvable while encoding a different initial state, goal, or optimization metric than the benchmark's curated reference. Conversely, some structurally exact reconstructions failed operational checks — up to 18.3% of instances on unsupported AutoPlanBench (Table 2) — a paradox the paper reports but does not analyze.
Which LLMs were used and did stronger models fix the gap?
Main results use gpt-4.1-mini; appendices add gpt-5.5 and opus-4.7. Stronger models reach near-perfect operational success on classical domains (0.983-1.000), but PDDL 2.1 structural match stays low (0.067-0.250), so reference reconstruction does not close even when executability does.
Is the LLM critic reliable as a faithfulness judge?
Only partially. In a 24-case manual audit, the critic agreed with human faithfulness judgments in 13/24 cases, with both false accepts (3) and false rejects (8). The authors themselves conclude it is useful as an operational feedback signal, not as an independent faithfulness oracle.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (pipeline overview) appears only as a caption in the extraction; the graphic itself is an image, so visual details of the loop could not be verified.
- Minor OCR artifacts in equations (e.g., summation notation in Section 4.4) and accented author names; all table values were legible and cross-checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.