DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculum
Haoqiang Kang, Yiming Zhang, Yiming Zhang, Yiyang Guo, Chuying Li, Jianzhi Shen, Tianruo Rose Xu, Xiaokang Ye, et al.
cs.LG
Quick verdict
DeliveryGym: courier-shift RL env with verifiable rewards; RL lifts Qwen3-VL-4B test income 54.3%, adaptive curriculum +18% vs uniform; one model trained.
Summary
DeliveryGym turns the DeliveryBench courier task into a full RL training environment: an Unreal Engine 5-based 3D city simulator in which agents accept, sequence, and deliver orders across continuous shifts, with time, energy, money, food quality, and safety persisting across deliveries and terminal rewards computed from simulator events. An adaptive curriculum diagnoses policy weaknesses (traffic compliance, obstacle recovery, multi-order planning) via fixed probes every 200 trajectories and retargets 70% of training task generation. On a fixed 130-shift suite across 13 maps, six frontier/open models fall far short of a search oracle — the best (GPT-5.6 Sol) sits 24% below despite 92% single-order success — and pixel-based Any-Point control cuts income 50–89% versus Waypoint control. GRPO training of Qwen3-VL-4B raises evaluation income 58% over initialization (13% over a 10k-trajectory SFT baseline); the step-100 safety-reward policy improves unseen-city test income by 54.3% while cutting violations. At equal rollout budget, the adaptive curriculum yields 18% higher evaluation income than uniform sampling (10% above a static curriculum). Scaling the task pool (to 144k configurations) and training maps (to 8) adds 47%/61% income at fixed trajectory budget. Limitations: only one 4B model is trained, in Waypoint mode with cached views; the oracle is not certified optimal; the listing abstract's 16.5% curriculum figure conflicts with the text's 18%; and no code release is explicitly stated.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across six models and 130 fixed shifts on 13 maps, reliable single-order execution does not imply profitable shift-level planning (best model 24% below the search oracle despite 92% single-order success).
- Online GRPO training of Qwen3-VL-4B raises unseen-city test income by 54.3% over initialization, and adding explicit safety penalties to the reward cuts violations while further increasing income.
- An adaptive curriculum that retargets 70% of generated training shifts from diagnosed weaknesses yields 18% higher evaluation income than uniform sampling at the same rollout budget.
- Scaling the available training task pool (to 144k configurations) and training maps (to 8) improves income by 47%/61% under a fixed trajectory budget.
Strengths
Best-in-class curriculum ablation design
Section 3.4/App. C compares adaptive sampling against uniform, a static curriculum frozen at the initial weakness profile, and random targeting — all sharing the same 70% targeted proportion, probe budgets, and optimizer steps — so the +18%/+10%/+14% deltas isolate how targeted practice is allocated, not merely that it exists.
Unusual verifier and cache transparency
App. E independently recomputes earnings in integer cents (999/1,000 agreement), replays task-state hashes (498/500), checks duplicate handoffs (200/200), and quantifies cached-observation bias on 130 paired shifts (mean −$0.10, p95 $0.80, max $3.00) instead of silently training on cached views.
Fixed-harness comparisons
Honest neutral/negative reporting in places
Decision-level diagnostics beyond headline income
Weaknesses
Single training model and scale
Abstract/body numeric inconsistency
Training confined to Waypoint with cached views
Missing uncertainty for core comparisons
Reproducibility rests on an in-house stack
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
DeliveryGym (arXiv 2609.19801v2, cs.LG; UCSD/JHU/Cornell plus an independent author) is an environment-and-training paper: it converts the DeliveryBench courier task into an online RL environment with (i) persistent world state across deliveries (time, stamina/battery, money, food condition, safety), (ii) verifiable trajectory rewards computed from simulator events (Eq. 1-2), and (iii) an adaptive curriculum that retargets task generation from diagnosed weaknesses (Section 2.5). Central claims: (1) six models on 130 fixed shifts across 13 UE5 maps show a gap between single-order execution and shift-level planning (Section 3.2, Fig. 3: GPT-5.6 Sol 92% single-order success but $18.3 vs the $24.0 oracle); (2) GRPO lifts Qwen3-VL-4B's unseen-city test income 54.3% over initialization, with safety reward terms cutting red-light violations 62% and collisions 54% while adding 14% income over earnings-only RL (Section 3.3, Fig. 4); (3) the adaptive curriculum beats uniform sampling by 18%, a static curriculum by 10%, and random targeting by 14% at 100 updates under matched budgets (Section 3.4, Fig. 5); (4) scaling the task pool to 144k configurations and maps to 8 adds 47%/61% income at fixed trajectories (Section 3.5, Fig. 6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DeliveryGym?
It is a reinforcement-learning environment for LLM/VLM agents: an Unreal Engine 5 based 3D city where an agent works continuous courier shifts — accepting, sequencing, and delivering orders — while time, energy, money, food quality, and safety persist across deliveries. Net income is computed from simulator events, so the reward is verifiable and usable for RL training, not just benchmarking.
How much did RL improve the agent in DeliveryGym?
Training Qwen3-VL-4B with GRPO raised evaluation income 58% over the untrained model (13% above a 10k-trajectory SFT baseline), and the step-100 safety-reward policy improved income on unseen-city test shifts by 54.3% while reducing red-light and collision violations. The adaptive curriculum added a further 18% over uniform task sampling at the same rollout budget.
Is the DeliveryGym code or dataset publicly available?
The paper references a project page (mk322.github.io/DeliveryGym) but the reviewed text does not explicitly state that the simulator, city maps, or evaluation suite are released. The system builds on the authors' in-house UE5 tooling (SimWorld Studio, SPEAR), so independent reproduction currently requires the authors' stack.
What are the main limitations of the DeliveryGym paper?
All training results use a single 4B-parameter model (Qwen3-VL-4B) in the simplified Waypoint movement mode with cached images; the visually harder Any-Point mode is evaluated but never trained. The search oracle is not a certified optimum, the listing abstract's 16.5% curriculum figure conflicts with the paper's 18%, and no human baseline is reported.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are text-extracted; error bars or intervals visible in figure graphics cannot be inspected, so judgments about missing uncertainty may partly reflect extraction rather than the PDF.
- v2 preprint with future-dated venues and models (ICLR 2026 citations, GPT-5.6 Sol, Claude Fable 5); citations and model identities could not be cross-verified from the text.
- The project page and any code release could not be checked from the provided text alone; codeAvailable=false reflects absence of an explicit statement.
- The task metadata abstract (16.5%) differs from the embedded v2 abstract (18%) on the curriculum figure, treated here as a version-level discrepancy.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.