Bayesian Optimization with Rich Auxiliary Information via LLMs
Tejus Gupta, Efe Mert Karag\"ozl\"u, Efe Mert Karag\"ozl\"u, Rohit Sonker, Barnab\'as P\'oczos, Barnab\'as P\'oczos, Jeff Schnieder
cs.LG
Quick verdict
LLM-elicited priors and auxiliary observations improve BO on HPOBench and fusion; promising, but LLM identity, hyperparameters, full-grid results unreported.
Summary
The paper studies how large language models (LLMs) can inject two forms of rich auxiliary information into Bayesian optimization (BO). First, prior elicitation: a function prior, obtained by asking the LLM to grade configurations into five ordinal quality bins (avoiding direct numerical prediction), RBF-interpolated into the GP mean; and an argmax prior, a Gaussian mixture over LLM-proposed maximizer locations that modulates the acquisition function πBO-style with 1/t decay. Second, auxiliary observations (training statistics, operator logs) appended to LLM prompts. On HPOBench (validation F1; SVM and neural network shown on German Credit and Vehicle Silhouettes, 100 seeds), both prior variants beat GP-UCB and LLM baselines (LLM-E2E, LLAMBO, LLINBO); LLM-E2E matches random search and LLAMBO shows high cross-task variability. On the contextual DIII-D tokamak task (281 historical shots, 20 seeds), an auxiliary-enhanced LLM-E2E variant substantially beats standard LLM-E2E, and auxiliary feedback also improves LLINBO on HPOBench (50 seeds). Limitations: quantitative figures cover only two model families and two datasets of the 5×5 grid; the underlying LLM is never identified; key hyperparameters (K bins, σ bandwidth, β, query counts) lack values and sensitivity analysis; no significance tests beyond bootstrap CIs; no code release stated.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLMs are substantially more effective in BO when modeling ordinal objective quality or distributions over promising inputs than when predicting objective values directly as surrogates.
- Injecting LLM-elicited function priors into the GP mean, or argmax priors into the acquisition πBO-style, outperforms GP-UCB and LLM-based baselines (LLM-E2E, LLAMBO, LLINBO) on the reported HPOBench tasks.
- Augmenting LLM optimization histories with auxiliary observations y_aux consistently improves performance across methods on both HPOBench and the DIII-D tokamak task.
- The value of LLMs in BO lies in injecting prior knowledge and side information into principled uncertainty-aware optimizers, not in replacing surrogates or acting as end-to-end policies.
Strengths
Well-motivated ordinal prior elicitation
Section 4.1.1 avoids the documented calibration failure of LLM numerics (citing [5,21,22]) by requesting one of K=5 ordered quality bins mapped to interval midpoints, then RBF-interpolating N Sobol samples into the GP mean; the GP-residual formulation plus data accumulation attenuates a bad prior naturally.
Transparent qualitative prior analysis
Figure 1 directly contrasts function vs argmax priors against true SVM F1 landscapes on four HPOBench datasets, showing function priors are noisy but roughly correct while argmax priors are smooth and near the optimum — the clearest evidence for the paper's core thesis.
Real-world scientific testbed with genuine side information
Auxiliary-observation gains across frameworks
Disclosed prompts and heavy seeding
Weaknesses
Quantitative evidence narrower than claims
The LLM is never identified
Fusion comparison under-substantiated
No significance testing; possible leakage confound
Fixed [0,1] bin intervals presuppose known objective range
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This method paper (CMU; cs.LG, arXiv:2609.19437) targets a real gap in Bayesian optimization (BO): real experiments produce far more than scalar evaluations — training curves, diagnostics, operator logs, and practitioner beliefs. The authors formalize two information channels: LLM-elicited *priors* (over function values, Sec 4.1.1, or over the maximizer x*, Sec 4.1.2), and *auxiliary observations* y_aux = g(x) appended to LLM context (Sec 4.2). Central claims: (i) LLMs are better ordinal/prior models than direct value predictors; (ii) function-prior GP means and argmax-prior acquisition modulation outperform GP-UCB and LLM-based baselines (LLM-E2E, LLAMBO, LLINBO); (iii) y_aux improves LLM-based optimization consistently across frameworks; (iv) LLMs belong in BO as prior/side-information interfaces, not surrogates or end-to-end policies. Claims (i)–(iii) are empirically supported within the reported scope — but the scope is narrower than the abstract implies.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How do the authors use LLMs in Bayesian optimization in this paper?
In three ways: (1) the LLM grades candidate configurations into five ordinal quality bins ('very poor' to 'very good') instead of predicting numbers, and the interpolated ratings become the Gaussian-process mean function; (2) repeated LLM proposals of likely optimum locations are fit as a Gaussian-mixture 'argmax prior' that modulates the acquisition function πBO-style, fading with a 1/t decay; (3) auxiliary data such as training metrics or operator experiment logs are added to the LLM's prompt alongside the usual (x, y) optimization history.
What benchmarks were used, and did the proposed methods win?
Two settings: HPOBench hyperparameter tuning (maximizing validation F1 over model families such as SVM and neural networks on OpenML datasets) and a contextual nuclear-fusion task using 281 historical DIII-D tokamak shots. In the shown experiments, both LLM-prior variants beat GP-UCB and LLM baselines (LLM-E2E, LLAMBO, LLINBO); LLM-E2E alone performed about as well as random search; and adding auxiliary observations consistently helped the LLM-based methods. The quantitative figures cover only two model families and two HPOBench datasets, however.
Is code or data available for this paper?
The text does not mention a code release. HPOBench is a public benchmark and sample prompts appear in an appendix, but the LLM used is never identified, key hyperparameters are not given numerically, and the DIII-D tokamak data come from a US national fusion facility, so full replication is not possible from the paper alone.
Do LLMs work as direct optimizers according to this paper?
No. The paper's LLM-E2E baseline performed comparably to random search, and LLAMBO (LLM as surrogate) showed high cross-task variability — strong on neural-network tasks but weak on SVM. The authors argue LLMs are most useful for supplying priors over promising regions and digesting side information, not for replacing the surrogate model or acting as the optimizer themselves.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure data (Figures 1–4) are available only via captions and in-text description; actual curve values, shapes, and CI widths could not be inspected.
- The LLM identity never appears in the extracted text; this is treated as a paper-level omission, though if it were stated inside a figure it could have been missed.
- arXiv metadata lists 'Jeff Schnieder' while the paper text reads 'Jeff Schneider'; the review assumes the in-text spelling.
- Appendix prompts illustrate an MLP/Adult task that does not appear in the reported experiments, so exact deployed prompts may differ.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.