Frontier LLMs are effective batch optimizers: Assessing reasoning models in continuous and discrete settings
Frank Hu, Shriram Chennakesavalu, David Graff
cs.LG
Quick verdict
Solid benchmark: LLMs match GP-EI on continuous tasks yet turn brittle under disguise; on PMO they are sample-efficient but ExLLM and GenMol still lead on AUC.
Summary
This preprint benchmarks three Anthropic reasoning models — Sonnet 4.6, Opus 4.8, and Opus 5 — as zero-shot batch optimizers in two regimes. On five classical test functions (Branin, Hartmann-3/6, Ackley, Rastrigin) with a budget of 120 evaluations and batch sizes q=1/2/4, Opus 5 multi-turn beats GP-EI on most task/batch combinations, but performance degrades with larger batches and under a domain bijection plus affine output disguise; disguised high-dimensional tasks remain unsolved. Exact-zero regret on plain Ackley/Rastrigin stems from the model recalling the function and emitting its analytic optimum, which the authors analyze openly (Fig. 2, App. A.3). On the 23 PMO molecular oracles at a 210-call budget, Opus 5 multi-turn is the strongest LLM condition (best on 17/23 oracles by Top-10 AUC, 20/23 by Top-10 mean) and competitive with most of a 33-method specialist panel — yet ExLLM (mean 0.738) and GenMol (AUC 0.500 vs 0.422) lead in aggregate, several oracles (sitagliptin_mpo 0.053, both isomer tasks) stay near zero, and LICO/GP-MoLFormer-SIM comparisons are budget-mismatched cross-paper numbers. Evidence rests on five seeds without hypothesis tests, and the scope is limited to one proprietary model family.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Frontier reasoning LLMs are competitive with or better than GP-EI as zero-shot batch optimizers on classical synthetic test functions, with Opus 5 multi-turn winning the majority of task/batch comparisons.
- LLM numerical optimization is brittle: performance degrades with larger batch sizes, higher dimensionality, and domain/output disguises, and plain-function successes partly exploit memorized function identities.
- On 23 PMO molecular oracles at a 210-call budget, Opus 5 multi-turn is the best LLM condition (Top-10 AUC best on 17/23, Top-10 mean on 20/23) and competitive with most of a 33-method specialist panel.
- Most specialist methods need roughly 3–40× more oracle calls to match the LLM's 210-call performance, although ExLLM and GenMol match or beat it with fewer calls.
- Multi-turn sessions consistently outperform single-turn prompting for optimization, indicating that persistent, stateful reasoning matters.
Strengths
Memorization-aware evaluation design
§4.1's disguise pipeline (axis permutation × Bernoulli flip × power warp, plus a LogUniform affine output map) preserves f(x*) while destroying function identity, and App. A.3 quantifies recall events — regret ratios drop orders of magnitude on plain tasks around function-name recalls (n = 3–9 events per task) — a rare, honest confound audit in this literature.
Budget-matched, fully crossed synthetic design
N=120 fixed across all methods with q∈{1,2,4}, single/multi-turn, three models, plain and disguised variants (Tables A2–A3). GP-EI's stability across batch sizes gives a clean reference exposing the LLM's drift (disguised Hartmann-3 MT: 1.00×10⁻⁶ at q=1 vs 1.14×10⁻² at q=4).
Broad, largely re-run PMO comparison
Calibrated, self-critical reporting
Weaknesses
Single-vendor scope behind a general title
Synthetic wins partly reflect memorization
Main-text showcase diverges from full-panel ranking
No hypothesis testing; n=5 on heavy-tailed oracles
Budget-mismatched cross-paper baselines and no cost accounting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether current-generation reasoning LLMs can serve as zero-shot *batch* optimizers, tested on two axes: (1) five classical continuous test functions (Branin, Hartmann-3/6, Ackley, Rastrigin) under a matched budget of N=120 evaluations with batch sizes q∈{1,2,4}, against GP-EI and random search (§4.1); (2) the 23 deterministic oracles of the PMO molecular-optimization benchmark at a reduced 210-call budget (10 init + 20×10), against a reproduced 33-method panel plus literature-reported LICO and GP-MoLFormer-SIM (§4.2, App. B). Three claims carry the paper: frontier LLMs are competitive-to-better than GP-EI on synthetic numerical tasks (Fig. 1: Opus 5 multi-turn leads 10/10, 8/10, 6/10 of comparisons at q=1,2,4); this competence is brittle with respect to batch size, dimensionality, and domain/output disguises; and in the semantically rich discrete setting the LLM is a strong, sample-efficient optimizer without scaffolding. The claims are mostly calibrated, but the synthetic-half claim leans partly on memorized optima, and the strongest discrete competitors (ExLLM, GenMol) beat the LLM on aggregate Top-10 AUC — a fact disclosed mainly in the appendix.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can LLMs like Claude Opus replace Bayesian optimization?
According to this paper, not fully. On five classic continuous test functions, Opus 5 in multi-turn mode beat a GP-EI Bayesian-optimization baseline in most task/batch settings, but its edge collapsed when functions were disguised to prevent recognition, and it emitted memorized analytic optima for plain Ackley/Rastrigin. On molecular optimization it was competitive at very small budgets (210 oracle calls), but the strongest specialist tools (ExLLM, GenMol) still achieved better aggregate scores.
What is the PMO benchmark?
Practical Molecular Optimization (Gao et al., 2022) is a suite of 23 deterministic 'oracle' functions that score small molecules — given as SMILES strings — on properties such as similarity to a target molecule, rediscovery of a known compound, and multi-property objectives. Methods are compared on how well they maximize these scores under a limited number of oracle evaluations, using metrics like Top-10 AUC and Top-10 mean.
How sample-efficient are LLM optimizers on molecules?
The paper measures how many extra oracle calls each baseline needs to match Opus 5's result at 210 calls. Most baselines — Bayesian optimization, genetic algorithms, REINVENT, random screening — needed roughly 3–40× more calls or never matched it, while the strongest learned methods matched or beat it with fewer calls: ExLLM at about 0.5× and GenMol between 0.14× and 2.26× across the four showcased tasks.
Did the LLM just memorize the test functions?
Partly, yes — the authors demonstrate this themselves. On undisguised Ackley and Rastrigin the model output the exact known optimum (zero regret), and analysis of reasoning traces (Appendix A.3) shows regret dropping sharply at turns where the model names the function. After disguising the domain and output range, performance on higher-dimensional functions degraded substantially, indicating that the undisguised results rely heavily on recognition rather than pure optimization.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- No code or data availability statement appears anywhere in the provided text; the paper does not state where prompts, trajectories, or the reproduced method panel can be obtained.
- LLM decoding parameters (temperature, top-p, number of completions per turn) are not stated in the provided text; only two representative opening-turn prompts are shown (Prompt 1 and Prompt 2).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.