PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving
Yuan Gao, Sebastian Müller, Mattia Piccinini, Marc Kaufeld, Yuchen Zhang, Finn Rasmus Schäfer, Qunying Song, Johannes Betz
cs.AI · cs.CL · cs.RO
Quick verdict
PlannerForge unifies scenario-based planner testing with LLM agents; broad 10-model×5-prompt eval, but stats thin and headline scores cherry-pick best cells.
Summary
PlannerForge is an LLM-agent framework that unifies the full scenario-based testing pipeline for motion planners in autonomous driving: generation from OpenStreetMap maps, retrieval from a CommonRoad database, modification, module routing, planner execution, and result analysis, plus LLM-driven planner cost enhancement and cross-planner benchmarking (Frenetix, MP-RBFN). Ten off-the-shelf backends (five commercial, five local 20-35B models) are evaluated under five prompt conditions with 200 queries per task (~80,000 calls). Best-per-task scores span 0.88 (Selection) to 1.00 (Planner Testing); chained end-to-end, the pipeline retains 83%/78% of 200 seed queries (commercial/open). It beats Scenario Factory 2.0 on executable generation (193 vs 144 of 200), BM25 on rank-1 selection (92.0% vs 67.5%), and From-Words-to-Collisions on physically valid edits (≥94% vs 31%), while running an order of magnitude slower. Cost tuning lifts Frenetix success from 50.4% to 70.2% and cuts collisions from 19.0% to 8.4% at N=400. Stated limitations: open-loop simulation, an unscored Analysis module, German-skewed data, Frenetix-only quantitative results; statistical testing is sparse and one reported Wilcoxon test is non-significant (p=0.44). Code and data are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PlannerForge is the first LLM-agent framework unifying all scenario-based testing stages (generation through assessment) plus planner cost enhancement and cross-planner benchmarking in a single chatbot pipeline.
- Off-the-shelf LLMs without domain-specific fine-tuning reach best-per-task scores of 0.88 to 1.00 across the five core tasks, and 20-35B open-source models match commercial APIs on most tasks.
- Chaining all modules end-to-end retains 83% (commercial qwen3.6-plus) and 78% (open qwen3.6:35b) of 200 seed queries under cp_icl_cot.
- PlannerForge beats Scenario Factory 2.0 on executable generation (193 vs 144 of 200), BM25 on rank-1 selection (92.0% vs 67.5%), and From-Words-to-Collisions on physically valid modification (>=94% vs 31%).
- LLM cost tuning lifts Frenetix success from 50.4% to 70.2% and cuts collisions from 19.0% to 8.4% at N=400 versus the hand-set Default configuration.
Strengths
Genuine unification claim backed by an unusually broad empirical grid
Covers all six Pegasus/SAKURA stages plus Enhancement and Benchmarking; evaluation spans 10 backends x 5 prompt conditions x 5 tasks at N=200/cell (~80,000 calls, Conclusion), with per-stage boolean scoring funnels (Tables 7, 9, 11-14, 16, 18) that separate semantic correctness from downstream simulability.
Honest cost accounting against classical baselines
The paper repeatedly concedes LLM overhead: generation 21.6 s vs 2.2 s for Scenario Factory 2.0 (Table 3); selection 21.6 s / 8.7k tokens vs <0.01 s / zero tokens for BM25 (Table 4); BM25 already hits 86.0% Any@5. End-to-end cost is ~126 s and ~42.5k tokens per scenario (Table 1).
Hallucination isolation by architecture, not just prompting
Strong reproducibility signals
Informative failure-mode analyses
Weaknesses
Sparse and partly contradictory statistical reporting
Best-per-task headline reporting invites cherry-picking concerns
Baseline comparisons carry favorable framings
The Analysis module is never scored
Collision-induction result admits two readings
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PlannerForge (TUM AVS; EMNLP 2026 main, arXiv:2609.08965) claims the first LLM-agent framework unifying the entire scenario-based testing lifecycle for motion planners: Generation from OSM maps via Overpass, a 500+-scenario CommonRoad database in ChromaDB, Selection, Test Execution, and ADS Assessment (the Pegasus/SAKURA six-stage taxonomy), extended with two LLM-era stages — planner cost Enhancement and cross-planner Benchmarking (Frenetix and MP-RBFN behind a shared πP interface). Central empirical claims: best-per-task scores 0.88–1.00 across ten off-the-shelf backends and five prompt conditions (~80,000 calls); 83%/78% end-to-end seed-query retention (commercial/open, Table 1); and wins over Scenario Factory 2.0 (193 vs 144 executable of 200, Table 3), BM25 (92.0% vs 67.5% Satisfy@1, Table 4), and From-Words-to-Collisions (≥94% vs 31% physical validity, Table 5). The novelty claim leans partly on the authors' own surveys (Gao et al. 2026b; Song et al. 2026).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PlannerForge?
An open-source LLM-agent framework from TU Munich (accepted to EMNLP 2026) that automates the entire scenario-based testing pipeline for autonomous-driving motion planners: it generates scenarios from real OpenStreetMap maps, retrieves and modifies scenarios from a CommonRoad database via natural language, runs planners (Frenetix, MP-RBFN) in SUMO-based simulation, analyzes results, and tunes planner cost parameters through a chat interface.
Does PlannerForge require fine-tuned or custom models?
No. It was evaluated with ten off-the-shelf LLMs — five commercial APIs (Qwen3.6-plus, Deepseek-v3.2, GLM-5, Gemini-3-flash, GPT-5.4-mini) and five local open-source 20-35B models served via Ollama on a single RTX 5090 — under five prompt conditions. The paper reports local models match commercial APIs on most tasks.
How does PlannerForge compare to classical tools in speed and cost?
It is much slower and costlier: about 21.6 seconds and 4.5k-8.7k tokens per query for generation/selection, versus 2.2 seconds for the rule-based Scenario Factory 2.0 and under 0.01 seconds with zero tokens for BM25 keyword search. End-to-end, one scenario costs roughly 126 seconds and 42.5k tokens on a commercial backend (99 s / 32.6k tokens open-source). In exchange it offers natural-language control and higher rank-1 precision (92.0% vs 67.5%).
What are the main limitations of the paper?
The simulation is open-loop (background traffic cannot react to the ego vehicle), the result-Analysis module has no ground-truth evaluation, quantitative planner results use only the Frenetix planner (MP-RBFN comparison is qualitative), the modification corpus is dominated by German maps, and most comparisons lack statistical significance testing — one reported Wilcoxon test is non-significant (p=0.44).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with two '[...sampling gap...]' regions (parts of Section 4.3, fragments of Sections 5.2/references, and portions of appendix narrative); every number cited in this review was verified present in the visible text.
- Figure-only content (e.g., the per-model score bars in Figure 5 and the Figure 8 panels) is described via captions and prose; underlying per-model values beyond the tables shown could not be independently checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.