Reinforcement learning to choose optimizers
Martin van der Schelling, Deepesh Toshniwal, Miguel A. Bessa
cs.NE · cs.LG · math.OC
Quick verdict
RL2CO learns via RL which optimizer to run and for how long; strong results, but a gap-maximizing test set and no prior-switch baselines temper the claims.
Summary
The paper introduces RL2CO, a reinforcement-learning framework that treats optimizer choice during a run as a sequential decision problem. At each decision point a recurrent policy reads the run state, selects one of four optimizers — L-BFGS, Rprop, CR-FM-NES, MR15-GA, spanning gradient-based and derivative-free methods — and decides how many iterations (from {10, 100, 1000}) it runs next. Switches transfer the incumbent best solution and a scalar search scale via per-optimizer 'handshake' contracts. A context proxy routes a gating network over recurrent expert heads; training uses decoupled actor-critic with PPO, and the reward is the drop of a potential based on the area under the convergence curve on a log evaluation axis. Training tasks (100 problems from noiseless, noisy, and high-dimensionally embedded BBOB) and the portfolio were jointly selected through a binary linear program ensuring no optimizer dominates. On 100 held-out problems the learned policy reaches a larger fraction of targets than every portfolio member at all but the smallest budgets, beats a random schedule, and stays robust on CEC 2005/2017/2013 plus three unseen families (quadratics, Gaussian classification, Adam meta-optimization). Main limitations: analytic benchmarks and small synthetic tasks only, no transfer across portfolios, iteration-denominated durations, and no direct comparison with prior switching methods.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- RL2CO is the first framework that learns at runtime which optimizer to use, for how long, and what state it inherits, over a portfolio spanning gradient-based and derivative-free methods.
- On 100 held-out BBOB problems, the learned policy reaches a larger fraction of targets than every portfolio optimizer at all but the smallest budgets, and beats a random schedule over the same portfolio.
- The policy is robust under distribution shift: it exploits selection signal on CEC 2005 and three unseen problem families, and degrades gracefully to its random-schedule baseline on CEC 2017/2013 where the exploitable signal is small.
- Jointly designing training tasks and portfolio via a constrained binary linear program (Eq. 5, ε=μ=0.02) guarantees no optimizer dominates, making algorithm selection non-trivial.
Strengths
Unified treatment of the three switching questions
Section 2.4 articulates the gap precisely: prior work fixes which/when/how and learns only one. RL2CO jointly learns optimizer choice, commitment duration (T={10,100,1000}), and state handover — none of the cited baselines ([10], [11-13], [14], [41-44]) does all three.
Principled joint task-portfolio construction
Appendix C.3 Eq. (5) selects 100 training problems and 4 optimizers from a pool of 3,024 BBOB instances (Table 4) and 52 candidates via a binary linear program maximizing the VBS margin, with an explicit no-dominance constraint (ε=0.02) and coverage constraint (μ=0.02), solved by exact enumeration of the 270,725 portfolios with LP bounds.
Fair control baseline and broad evaluation
Honest reporting of failure regimes
Reproducibility package
Weaknesses
Test set optimized for the method's sweet spot
No head-to-head with published dynamic-switching methods
No inferential statistics or seed variance
Iteration-denominated commitments vs evaluation-denominated metric
Scale of evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper formulates mid-run optimizer switching as a contextual MDP (“RL2CO”). At each decision a recurrent policy picks one of four portfolio optimizers (L-BFGS, Rprop, CR-FM-NES, MR15-GA — two gradient-based, two derivative-free) and a commitment duration from {10, 100, 1000} iterations; switches transfer the incumbent best solution, its value, and a scalar search scale σ via per-optimizer “handshake” contracts (Tables 1–2). Section 2.4 states the knowledge gap crisply: no prior method jointly learns which optimizer, how long it runs, and what state it inherits. The central claims are (i) the learned policy beats every portfolio member on 100 held-out BBOB problems at all but the smallest budgets (only below roughly one evaluation per dimension does L-BFGS lead, Fig. 5); (ii) the margin over a random schedule attributes the gain to informed decisions rather than switching itself; (iii) robustness under distribution shift on CEC 2005/2017/2013 and three unseen families (Figs. 7–11).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does RL2CO do?
RL2CO is a reinforcement-learning policy that, during a single optimization run, repeatedly chooses which optimizer from a small portfolio (L-BFGS, Rprop, CR-FM-NES, MR15-GA) runs next and for how many iterations, transferring the best solution found so far and a scalar search scale between optimizers at each switch.
Does the learned policy beat every individual optimizer?
On 100 held-out BBOB benchmark problems it reaches a larger fraction of targets than every portfolio member at all budgets except the smallest (below roughly one evaluation per dimension, where L-BFGS leads). On some shifted suites such as CEC 2017 and CEC 2013 it only matches a random schedule over the same optimizers.
Is code available for this paper?
Yes. The authors state that code, dataset, and documentation follow FAIR principles and are available at github.com/bessagroup/rl2co, alongside a JAX implementation of the BBOB functions (bbob-jax, archived on Zenodo).
What are the main limitations of RL2CO?
Evaluation stays on analytic benchmarks and small synthetic learning tasks; the policy does not transfer across portfolios (changing optimizers requires retraining); commitment durations are counted in iterations rather than function evaluations; and no prior dynamic-switching methods are directly compared in the experiments.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure contents (ERTD curves, per-realization schedule strips, training curves) appear in the text only as captions; their plotted values cannot be inspected, so curve-level quantitative judgments rely on the authors' descriptions.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.