Replan, Repair, or Edit? A Unified Empirical Evaluation of Travel Agents for Itinerary Revision under Resource Disruptions
Xiaofei Yuan, Yan Zhang, Shaobo Qiao, Huangleshuai He, Leyan Ni, Mingchen Ju, Lujia Yang, Sijia Xu, et al.
cs.AI · cs.MA
Quick verdict
Compares replanning, HTN repair, and LLM editing on TREK disruptions; a careful unified study, though evidence rests on adapters without statistical tests.
Summary
This paper presents a unified empirical comparison of three paradigms for revising accepted travel itineraries after resource disruptions: full replanning (an LLM–Z3 adapter using one LLM call plus Z3 constraint solving), classical hierarchical plan repair (IPyHOPPER, an HTN method requiring no LLM inference), and LLM-based local editing (an iTIMO adapter performing add/replace/delete operations on points of interest). Two TREK-derived benchmark sets — 500 single-disruption cases (375 feasible plus 125 infeasible required-hotel conflicts) and 200 feasible simultaneous compound-disruption cases (50 each of flight–hotel, flight–attraction, hotel–attraction, and all three) — are run through a common revision interface and feasibility criterion. LLM–Z3 with Gemini achieved the highest compound-disruption success (93.0%), and its single-disruption overall success (97.0%) essentially tied IPyHOPPER (96.8%). Yet successful IPyHOPPER and iTIMO repairs made about one edit (about two for compound cases) and preserved roughly 86–93% of accepted slots, versus 6.6–9.1 edits and 57–65% preservation for replanning. Costs diverged sharply: IPyHOPPER used no LLM calls, LLM–Z3 one compact call, and the iTIMO adapter 19,486–67,631 tokens per case. Main limitations: findings hold for specific adapters rather than paradigms, there is no statistical significance testing or confidence intervals, compound groups use unmatched itineraries, some error records were selectively rerun, Qwen iTIMO lacks raw records, and no code release is mentioned.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under a common TREK-derived revision benchmark, LLM-Z3 with Gemini achieved the highest observed compound-disruption success (93.0%, 186/200).
- IPyHOPPER nearly matched LLM-Z3-Gemini's single-disruption overall success (96.8% vs 97.0%) while preserving about 93% of accepted slots versus about 65% for full replanning.
- Successful hierarchical and local repairs made far fewer logical edits (about 1 single-disruption, 2.1-2.2 compound) than full replanning (6.6-9.1).
- Computational profiles differ sharply: IPyHOPPER uses no LLM inference, the evaluated LLM-Z3 adapter one compact call, and the iTIMO adapter 19,486-67,631 tokens per case.
- No strategy dominates all dimensions, and the selection guidance is explicitly limited to the evaluated implementations and synthetic disruption setting.
Strengths
A genuinely unified evaluation interface that closes a real comparability gap
Section 2.2 (Eq. 2) defines one revision contract f(q,P,Delta,K') -> P' / bottom / fail with explicit failure semantics; Section 4.1 gives all methods identical fixed instances, hides reference repairs and feasibility labels, and counts errors/timeouts as failures in the denominators rather than letting them vanish.
Unusually honest claim calibration
The abstract and Section 5 hedge with 'observed' and 'within evaluated settings'; Section 4.2 explicitly declines to rank the one-case gap 485/500 vs 484/500; Section 5 unpromptedly lists confounds (feedback, candidate restrictions, model routing, budgets) and states no method returns the minimum-edit optimum of Eq. 3.
Stability and cost metrics beyond bare success rates
Paired joint-success analysis for stability
Transparent data provenance
Weaknesses
Paradigm-level framing over adapter-level evidence
No statistical inference anywhere
Narrow infeasibility coverage
Compound-disruption groups are not difficulty-matched
No artifact release and one unverifiable data source
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Peer Review: Replan, Repair, or Edit? (arXiv 2609.19654)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper compare?
It compares three ways to fix an accepted travel itinerary after disruptions such as flight cancellations, hotel unavailability, or attraction closures: full replanning with an LLM plus the Z3 constraint solver (LLM-Z3), classical hierarchical HTN plan repair (IPyHOPPER), and direct LLM-based editing (an iTIMO adapter), all evaluated on 700 cases derived from the public TREK benchmark under one common interface and feasibility criterion.
Which revision strategy is best?
There is no single winner. For maximum feasibility recovery, LLM-Z3 with Gemini was strongest (93% success on compound disruptions). For preservation at zero LLM cost, IPyHOPPER nearly matched single-disruption success (96.8% vs 97.0%) while keeping about 93% of the accepted itinerary, though with a 300-second latency tail. The LLM editing adapter (iTIMO) preserved similarly but consumed 19,000-68,000 tokens per case and had the lowest compound-disruption coverage (67-75.5%).
Is code or data available for this paper?
The paper does not mention any code or benchmark release. The instances derive from the public TREK framework, but the disruption benchmark sets and the three adapters are not stated as publicly available, so exact replication currently depends on the authors.
What are the main limitations of the study?
Conclusions apply to the specific adapters tested rather than paradigms in general; there are no statistical significance tests; compound-disruption groups use different, difficulty-unmatched itineraries; some error records were selectively rerun; Qwen iTIMO figures lack raw logs; and the infeasible cases cover only required-hotel conflicts, so refusal ability is tested narrowly.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are diagrams whose content is inferred from captions and surrounding text only.
- Model names are service aliases (e.g., 'Gemini 3.8 Flash'); exact model versions and endpoint behavior cannot be verified from the text.
- No appendix is present, so prompt templates, per-case records, and benchmark-construction code cannot be inspected from the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.