Code-to-Harness: Distilling Black-Box Optimizers from Self-Play
Yi Wu, Zheng Ren, Zhiyu Hu, Haochen Wang, Daryl Chang, Li Wei, Ting Wang, Zhen Li, et al.
cs.LG
Quick verdict
Distills LLM practice into a frozen text harness cutting black-box regret ~48%; methodologically careful but narrow scope and unverifiable production claims.
Summary
This method paper proposes a practice–gate–distill–freeze pipeline: an LLM agent iteratively writes optimizer programs on development objectives (shifted positive-semidefinite quadratics), retains a champion through paired challenger evaluations, then one-shot distills the winning program and its practice record into a frozen ~197-word natural-language harness. On [-5,5]^8 problems with a 20-evaluation budget, the harness cuts Gemini Flash final simple regret from 69.5 to 36.0 (48%, p<.001) in an independent N=30 study, transfers unchanged to Flash-Lite and Claude Sonnet, and moves the model into the GP-BO performance tier on held-out BBOB landscapes, though it does not beat the retained program or strong GP-BO. An independent replication produced a different program and text (Harness B) at the same tier, and one gate-failed candidate is disclosed. A sealed YouTube reward-tuning case study shows the lowest point-estimate regret versus Vizier, but reports only 30 bootstrap resamples of one internal dataset with no significance claims. Limitations: a single budget (20) and dimension (8), many N=10 cells with wide intervals, and one key ablation lacking released trajectories.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A once-distilled, frozen ~197-word text harness reduces Gemini Flash final simple regret by 48% (69.5→36.0, N=30, p<.001) in low-budget black-box optimization.
- The unchanged text transfers across Gemini executors and to Claude Sonnet (43–49% regret reductions, p≤.005), showing the policy is portable language rather than a model-specific artifact.
- An independent end-to-end replication yields a different program and a different text (Harness B) at the same performance tier, supporting behavioral rather than string-level reproducibility.
- On held-out BBOB landscapes the harness moves the LLM into the GP-BO performance tier but does not dominate classical optimizers after multiple-comparison correction.
- On a sealed YouTube reward-tuning benchmark the framework attains the lowest point-estimate regret versus Vizier and GP-BO, reported descriptively without significance claims.
Strengths
Pre-sealed, leak-resistant evaluation protocol
Section 3-4: text and execution config are sealed before held-out/production evaluation; Appendix D.2's exact-solve audit (0/30 text-only, 0/10 sandboxed Rastrigin) credibly rules out benchmark identification shortcuts.
Powered confirmatory study for the central claim
The independent N=30 study (Section 5.1, Appendix G) shows 69.5→36.0 (48%, p<.001) with a percentile-bootstrap 95% CI of [19.0, 49.7] and 27/30 paired instances improving — not a retrospective enlargement of the N=10 primary cells.
Well-chosen controls against trivial explanations
Behavioral reproducibility actually tested
Candid statistical framing
Weaknesses
Narrow operating regime
Harness does not beat its own program or strong BO
Key mechanism ablation is provisional
Production evidence is unverifiable
Fragile N=10 cells underpin most secondary claims
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether an LLM agent can acquire a numerical search policy through executable practice and then deploy it as frozen natural language. It contributes (i) a practice–gate–distill–freeze framework (Section 3) and (ii) an evaluation protocol featuring pre-sealed artifacts, an independent N=30 powered study, an end-to-end replication (Harness B), and a disclosed gate-failure (Candidate C). Central claims: Harness A cuts Gemini Flash final simple regret 48% (69.5→36.0, p<.001, N=30); the unchanged text transfers across Gemini executors and to Claude Sonnet; on held-out BBOB the LLM enters the GP-BO tier without dominating classical baselines; and the framework attains the lowest point-estimate regret on a sealed YouTube reward-tuning benchmark (Table 12: 0.0495 vs Vizier 0.0894 at 20 evals).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a 'harness' in this paper?
A short frozen natural-language strategy text (197–205 words) that an LLM agent distilled from its own practice of writing and testing optimizer programs. At deployment it is prepended to an LLM executor to improve its black-box optimization behavior, with no further updates allowed.
Does the distilled harness beat Bayesian optimization?
No. It moves the LLM into the GP-BO performance range, but on the practice family the retained program (17.1) and strengthened GP-BO (28.7) still beat the harness (32.9), and on held-out BBOB no comparison against BO baselines survives multiple-comparison correction. The paper's claim is tier transfer, not dominance.
Does the harness work with models other than Gemini?
Yes, in one cross-provider test: the same frozen text transferred to Claude Sonnet, reducing regret by 43% on quadratics (83.5→47.3) and 49% on shifted Rastrigin (106.6→54.0), both at p≤.005 in the N=30 study.
Is the YouTube production result statistically significant?
No. The authors evaluate on 30 paired bootstrap resamples of one sealed historical dataset, which are not independent trials, and explicitly report point estimates only (0.0495 vs 0.0894 for Google Vizier at 20 evaluations) without hypothesis tests or superiority claims.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–4 are present only as captions in the extracted text; plotted curve data (convergence curves, per-executor intervals in Figure 2a) could not be verified numerically.
- Per the paper's own notes, several result cells (GPT-5 runtime cells, Gemini-Pro code-only ablation, production replicates) are reported as-is from unreleased study records; their numbers cannot be cross-checked against released caches.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.