JumpStart Your Policy Learning with Lessons from 160,000 Training Runs
Nabil Omi, Eric Bae, Chung Yik Edward Yeung, Siddhartha Sen, Ali Farhadi
cs.LG
Quick verdict
Massive 162,864-run study: tuning, benchmark mix, and data source reshape offline-RL rankings; rigorous, openly released; normalization choices merit scrutiny.
Summary
The authors present JumpStart, a large-scale empirical study of offline reinforcement and imitation learning. They train 162,864 policies covering ten algorithms (BC, BCP, BCQ, CQL, DT, IQL, ReBRAC, ACT, VQ-BeT, Diffusion Policy) across 114 Minari datasets — 57 continuous-control and 52 discrete-control (Atari) after filtering — reporting roughly one million GPU-hours. Using IQM and optimality-gap aggregates with bootstrap confidence intervals, they find no dominating algorithm: ACT leads the continuous aggregate and BCP the discrete, but leaders change across environment collections and expert sources, whose rankings transfer poorly (mean Kendall's tau about 0.16-0.17, below a count-matched random baseline). Hyperparameter tuning frequently reshuffles rankings (IQL overtakes BCQ after tuning), hyperparameter variability exceeds seed variability in 92% of algorithm-dataset cells, and 10-15-dataset benchmarks yield a different winner than the full suite in roughly 20-25% of cases. The paper derives median-transfer defaults that raise IQM from 0.47 to 0.70 against d3rlpy defaults on 25 held-out environments, and a dataset-conditioned random-forest recommender improving top-3 overlap from 0.409 to 0.485 under leave-one-environment-family-out validation. Limitations include heavy-tailed normalized scores (up to 11.13) entering aggregates, seed-based validation of the top-quartile approximation on continuous control only, and a recommender trained on just 57 datasets.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No single offline policy-learning algorithm dominates: aggregate leaders (ACT on continuous control, BCP on discrete) change across environment collections and expert sources, whose rankings transfer poorly (mean Kendall's tau ~0.16-0.17, below a count-matched random baseline).
- Hyperparameter tuning frequently reshuffles perceived algorithm rankings (IQL overtakes BCQ after tuning in Fig. 1), and hyperparameter variability exceeds seed variability in 92% of algorithm-dataset cells.
- Benchmark subsets of 10-15 environment-dataset pairs — sizes comparable to established suites — produce a winner conflicting with the full suite in roughly 20-25% of random draws, and carefully composed subsets of similar size can place ACT, DT, or BCQ first.
- Coordinate-wise medians of top-10% trials yield transferable default configurations that recover roughly half the benefit of full per-dataset tuning (IQM 0.47 to 0.70 on 25 held-out environments vs d3rlpy defaults).
- A dataset-conditioned random-forest recommender using observable dataset features improves top-3 recommendation overlap from 0.409 (global-leaderboard baseline) to 0.485 and reduces best-of-3 regret from 0.091 to 0.030 under leave-one-environment-family-out validation.
Strengths
Unprecedented scale
162,864 trained policies over ten algorithms and 114 datasets (Table 2), roughly one million GPU-hours — roughly two orders of magnitude beyond the closest prior sweep (Paine et al., 2,560 policies; Table 1), which lets the authors measure ranking instability as a distribution rather than an anecdote.
Rigorous aggregate statistics
Follows Agarwal et al. throughout: IQM and optimality gap with stratified bootstrap CIs (Figs. 8, 13, 18, 20), Wilson intervals for subset-winner probabilities (Fig. 11), Kendall's tau for rank agreement, and Brier/calibration metrics for the recommender's confidence output (Fig. 24) — a standard most prior offline-RL comparisons lack.
Self-auditing analyses
Actionable, validated deliverables
Full artifact release
Weaknesses
Normalization convention yields extreme scores feeding all aggregates
Leaderboard estimator is selection-based, validated on continuous control only
Simulator-only, single-ecosystem coverage
Shipped recommender is not the most accurate validated model
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
JumpStart (arXiv 2609.13730) is a large-scale empirical study of offline policy learning — offline RL plus imitation learning — built on 162,864 trained policies from ten algorithms (BC, BCP, BCQ, CQL, DT, IQL, ReBRAC, ACT, VQ-BeT, Diffusion Policy) across 114 Minari datasets: 57 continuous-control and 52 discrete-control (Atari) after filtering degenerate datasets (Sec. 3.1, Table 2), at a reported cost of nearly one million GPU-hours. Central claims: (1) no algorithm dominates — ACT tops the continuous-control aggregate and BCP the discrete one (Fig. 13), yet leaders flip across environment collections (Fig. 14) and expert sources (Fig. 12); (2) hyperparameter tuning reshuffles perceived rankings — after tuning, IQL overtakes BCQ and BC becomes far more competitive than in prior leaderboards (Fig. 1; Sec. 2); (3) benchmark composition matters — subsets of 10-15 environment-dataset pairs, the size of established suites, pick a winner conflicting with the full suite in 24.7%-15.45% of random draws (Fig. 11), and composed subsets of 10-12 pairs can place ACT, DT, or BCQ first (Fig. 9); (4) simple median-of-top-trials defaults transfer across environments; (5) a dataset-conditioned recommender beats a global-leaderboard baseline (Fig. 20).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the JumpStart paper about?
It is a large-scale empirical study of offline reinforcement and imitation learning. The authors trained 162,864 policies with ten algorithms across 114 datasets (about one million GPU-hours) to measure how hyperparameter tuning, benchmark composition, and dataset source change which algorithm looks best. They also release all trained policies, per-run scores, hyperparameters, code, and an extensible leaderboard.
Which offline RL algorithm should I use for my task?
The study finds no universal winner: ACT leads the continuous-control aggregate and BCP the discrete-control aggregate, but the best algorithm changes across environment collections and data sources. The authors release a dataset-conditioned random-forest recommender that suggests three algorithms from observable dataset properties; it recovered 48.5% of the true top-3 under validation, versus 40.9% for simply picking the globally strongest methods.
How many hyperparameter trials are needed for a fair offline RL comparison?
The paper's tuning curves show most performance gains appear within 25-50 trials, but stable algorithm rankings take longer: in continuous control, agreement with the full-budget ranking improves gradually even after scores plateau. Across the full sweep, hyperparameter choice caused more variation than random seeds in 92% of algorithm-dataset cells.
Is the JumpStart corpus and code publicly available?
Yes — the paper states it releases every trained policy, per-model scores and hyperparameters, training and evaluation code, strong baselines for all environments, and an extensible website/leaderboard that accepts externally contributed results. The abstract links to code, models-and-data, and the website.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with marked gaps: part of Section 3 (training/tuning setup between Table 2 and Table 3), the tail of Section 4.3 (discrete-control leaderboard discussion after Fig. 13), the end of Section 6 through the opening of Section 7, and appendix material between the Figure 25 sensitivity pages and the reference list are not visible. Judgments above rest on the visible text; no claim here depends on the skipped regions.
- Link targets for Code / Models and Data / Website appear only as labels in the extracted text; concrete repository URLs are not visible, so codeAvailable reflects the paper's stated release rather than verified artifacts.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.