Which LLM is Best for Translating Natural Language Goals to PDDL
Tomas Balyo, Lukas Chrpa, G. Michael Youngblood
cs.AI
Quick verdict
Claims all six frontier LLMs exceed 92% at NL-to-PDDL goal translation with Gemini 2.5 Flash best (96%); plausible but single-run, n=90 results lack statistical support.
Summary
The paper reports an empirical comparison of six frontier LLMs (OpenAI o3 and GPT-4.1, Gemini 2.5 Pro and Flash, Claude Opus 4 and Sonnet 4) on a narrow but practically motivated task: translating informal natural-language test goals from video game QA into PDDL goal specifications, or explaining why a goal cannot be expressed in the given domain. Using a single, iteratively refined prompt template (full domain PDDL plus a typed object list instead of the whole problem file, inexpressibility examples, and a conciseness directive), the authors evaluate 90 goals, 45 expressible and 45 not, across eight domains (four IPC-standard, four game-inspired), authored largely by PDDL-naive company employees. All models exceed 92% correctness: Gemini 2.5 Flash leads at 96% with the fewest false positives, outperforming Gemini 2.5 Pro at lower latency, while GPT-4.1 is fastest (1.61 s average) but least accurate; only one of 540 responses had invalid syntax. Expressible goals were scored by exact match or expert review; inexpressible cases were auto-graded on refusal alone, without auditing explanation quality. Key limitations: each model ran once per goal, inter-model differences amount to 2-4 items on n=90 with no significance testing; Table 1 correctness rates do not reconcile with the reported FP/FN counts for several models; the prompt appears refined on the same benchmark; and no code or domain models are stated to be released.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under one fixed prompt template, all six evaluated frontier LLMs correctly translate or correctly refuse more than 92% of 90 realistic game-testing goals.
- Gemini 2.5 Flash is the most accurate model (96%) with the fewest false positives (2), outperforming Gemini 2.5 Pro while being roughly three times faster.
- GPT-4.1 is the fastest model (1.61 s average response) but the least accurate (92%).
- Specific prompt choices, listing typed objects instead of full problem files, adding inexpressibility examples, and demanding concise goals, measurably raise accuracy and reduce false positives.
- Remaining failures stem chiefly from natural-language ambiguity and limits of domain representation rather than syntax (only 1 invalid-syntax response in 540 calls).
Strengths
Realistic, bias-controlled goal authorship
The 90 benchmark goals were written by company employees, more than half with no PDDL experience and without access to the domain models (Benchmarks section), avoiding the expert-written or LLM-generated bias common in text-to-PDDL datasets.
Treats inexpressibility as a first-class outcome with separable error types
The 45/45 expressible/inexpressible split and separate FP/FN accounting capture the practically dangerous failure mode: silently emitting a PDDL goal for an unachievable request (Gemini 2.5 Flash had the fewest, 2).
Transparent and instructive prompt-design narrative
Controlled breadth and near-solved format compliance
Practically actionable headline
Weaknesses
No statistical rigor behind the model ranking
Table 1 does not internally reconcile
Asymmetric evaluation strictness
Overfitting and conflict-of-interest risk in benchmark construction
Incomplete reporting for practical decision-making
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which LLM is best for translating natural language goals into PDDL?
In this study, Google's Gemini 2.5 Flash scored highest: 96% of 90 game-testing goals were handled correctly (translated to PDDL or correctly flagged as inexpressible), with the fewest false positives. Gemini 2.5 Pro and Claude Opus 4 scored 94%, o3 and Claude Sonnet 4 93%, and GPT-4.1 92%, though GPT-4.1 was the fastest at 1.61 seconds average. The spread between models is only 2-4 goals and was not tested for statistical significance.
What benchmark did the paper use?
90 natural-language goals written by game-industry employees, split evenly into 45 that can be expressed as PDDL goals and 45 that cannot, across eight planning domains: four International Planning Competition domains (Rovers, Barman, Woodworking, Parking) and four game-inspired domains (a first-person shooter, two real-time strategy games, and a city-building adventure).
What is a false positive versus a false negative in this paper?
A false positive occurs when the LLM outputs a PDDL goal for a request that fundamentally cannot be expressed in the given domain; this is dangerous because a downstream planner would silently pursue an unachievable goal. A false negative is when the LLM refuses to produce a goal for a request that actually is expressible, which is wasteful but safer. Gemini 2.5 Flash had the fewest false positives (2).
Is the code or dataset from this paper available?
The paper prints all 90 natural-language goals in its appendix and gives the complete prompt template, but it states no public release of code, domain PDDL files, object lists, or reference goals, so exact replication currently requires reconstructing the eight domain models yourself.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2 runtime distributions cannot be analyzed from the extracted text; only the average response times in Table 1 were used.
- The domain PDDL files, object lists, and reference goals are not included in the text, so the semi-automated correctness judgments cannot be independently verified.
- Appendix goal tables may contain line-merging artifacts from extraction, so exact per-domain goal counts could differ slightly.
- The discrepancy between correctness rates and FP/FN/IS counts could stem from an unreported column in Table 1 rather than an authoring error.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.