UTP-Bench: Uncertainty-aware Travel Planning Benchmark
Etcharla Revanth Rao, Priyanshu Karmakar, Shubhojit Mallick, Manish Gupta, Shreya Ghosh, Abhik Jana
cs.AI · cs.CL
Quick verdict
UTP-Bench adds empirical delay/crowd uncertainty to LLM travel-plan evaluation; promising, but annotation-metric circularity and no stats tests weaken claims.
Summary
The paper introduces UTP-Bench, a benchmark for evaluating LLM travel-itinerary generation under real-world uncertainty: 1,000 queries for 3-, 5-, and 7-day trips across 504 Indian cities, each paired with a gold itinerary from 14 trained annotators. Building on TripCraft's constraint framework, it adds empirical transit-delay distributions (FlightStats for flights, multi-horizon train delays from Tripozo, TomTom road delays) and BestTime crowd-density data for 20 major cities, plus three risk profiles (tolerant, optimized, averse). Three new metrics — Buffer Adequacy Score (BAS), Crowd-Aware Timing Score (CATS), and Transport Delay Absorption Score (TDAS) — complement five TripCraft metrics. Evaluating GPT-5, Qwen3-14B, Mistral-7B-Instruct, and Phi-4-mini, delivery rates stay at 92–100% while final pass rates collapse toward zero at 7 days; GPT-5 leads micro-CPR (88.86% at 3 days), all models score TDAS of at most 11.85%, and CATS (47–64%) proves easiest. The authors report human plans degrading from 91% to 63% between 3- and 7-day horizons. Main limitations: the annotation guidelines prescribe the same buffer multipliers the metrics enforce, risking circularity; no significance tests or decoding details are reported; Table 16's dataset counts conflict with Table 1; and the data rests on commercial APIs.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing travel-planning benchmarks (TravelPlanner, TripCraft, ChinaTravel) assume deterministic environments and therefore miss plan robustness under real disruptions.
- UTP-Bench is claimed as the first benchmark to evaluate travel plans against empirical transit-delay and crowd-density distributions, with risk-aware traveler profiles.
- Current LLMs (GPT-5, Qwen3-14B, Mistral-7B, Phi-4-mini) generate qualitatively coherent but fragile itineraries: 92–100% delivery yet near-zero final pass at 7 days and TDAS never above 11.85%.
- The three proposed metrics (BAS, CATS, TDAS) reveal model-specific failure patterns that binary pass-rate evaluation cannot distinguish.
- A large gap separates LLM plans from human gold itineraries — and even human plans degrade from 91% to 63% as the horizon grows from 3 to 7 days.
Strengths
Real, empirically grounded gap vs prior deterministic benchmarks
§2 positions UTP-Bench against TravelPlanner/TripCraft/ChinaTravel (deterministic) and TripTide (synthetic disruptions); §3.1 and Appendix A ground uncertainty in real distributions — 68,068 train records with multi-horizon delays, FlightStats on-time stats, TomTom 4,052 road-delay routes, BestTime crowd tiers for 20 cities (Table 1).
Unusually complete metric specifications
Appendix B (Eqs. 8–30) gives full inputs, formulas, and parameter tables for BAS/CATS/TDAS — restaurant base durations by meal type (Table 9), congestion factors Cf (Table 10), risk multipliers (Tables 11/15), and CATS quiet-bonus/peak-penalty mechanics (Eqs. 14–20) — enabling independent reimplementation.
Uncertainty metrics expose failure modes binary pass rates hide
Deterministic ablation independently motivates the benchmark
Scale and source transparency
Weaknesses
Metric-annotation circularity inflates the human-vs-model gap
No statistical rigor anywhere
Unexplained internal data contradiction
GPT-5 is both query generator and flagship evaluatee
Human baseline under-specified and annotator pool narrow
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
UTP-Bench (arXiv 2609.02421; IIT Bhubaneswar with Microsoft India) attacks a genuine gap: TravelPlanner, TripCraft and ChinaTravel score itineraries in deterministic worlds, whereas real trips confront transit delays and crowd swings. The paper claims three contributions (§1): a 1,000-query benchmark spanning 3-, 5- and 7-day itineraries over 504 Indian cities with human gold plans; an empirical uncertainty layer (delay distributions, crowd densities, three risk profiles); and three new metrics — BAS, CATS, TDAS. The headline empirical claim (§5, Observations 2–3): all four tested LLMs (GPT-5, Qwen3-14B, Mistral-7B-Instruct-v0.3, Phi-4-mini-Instruct) produce coherent but fragile plans — delivery 92–100% while Final Pass collapses to 0.00% micro at 7 days (Table 3), TDAS never exceeds 11.85% (GPT-5, Riskav, 7-day), and human plans fall from 91% to 63% as the horizon grows. The hedged priority claim — 'first benchmark to explicitly model stochastic robustness' (§2) — is defensible against TripTide's synthetic disruptions but must always be read alongside that prior work.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is UTP-Bench?
A benchmark for evaluating whether LLM-generated travel itineraries remain robust under real-world uncertainty. It contains 1,000 travel queries for 3-, 5-, and 7-day trips across 504 Indian cities, each with a human-annotated gold itinerary, and scores plans with three uncertainty-aware metrics (BAS for time buffers, CATS for crowd-aware timing, TDAS for transport-delay absorption) plus five constraint/persona metrics carried over from TripCraft.
How does UTP-Bench differ from TravelPlanner?
TravelPlanner evaluates static constraint satisfaction in a deterministic world (fixed schedules, no delays). UTP-Bench layers empirical transit-delay distributions (flights, trains, buses, cabs) and hour-by-hour crowd-density data for attractions/restaurants over a similar constraint framework, adds traveler risk profiles (tolerant/optimized/averse), and is India-specific (504 cities).
What did the UTP-Bench experiments find?
Across GPT-5, Qwen3-14B, Mistral-7B-Instruct, and Phi-4-mini, models deliver plans 92–100% of the time but almost none survive full evaluation at 7 days (final pass ≈ 0%). Avoiding crowds (CATS, 47–64%) is relatively easy; allocating buffers that absorb realistic transport delays (TDAS, max 11.85%) and calibrating buffers to risk profiles (BAS, mostly 1–12% for GPT-5/Qwen3) is very hard. Human plans also degrade with horizon (91%→63%) but stay well above models.
Is the UTP-Bench code and data available?
The paper links a public GitHub repository (github.com/ETCHARLAREVANTHRAO/UTP-Bench) and fully specifies its metrics. However, the underlying data was assembled from commercial APIs (Tripozo, BestTime, TomTom, FlightStats, TripAdvisor, Numbeo), so redistribution rights and full data release are not clearly guaranteed in the text.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Equation (3) in §4 renders with garbled fraction layout in the PDF extraction; the equivalent, cleanly extracted formulas in Appendix B (Eqs. 8–13, 27–30) were used for the metric assessment.
- Figure-only content (Fig. 2 crowd distributions; Figs. 10–12 annotator demographics) is not present as text; no claims in this review rely on those figures.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.