Online Surrogate Repair: Decoupling High-Fidelity Feedback from Search Length in Closed-Loop Discovery
Xiaotang Feng, Philip Torr, Bruno Andreis
cs.LG · cond-mat.mtrl-sci · cs.AI
Quick verdict
OSR sparsely repairs an in-context surrogate during agent search; large consistent regret gains, but overlapping CIs and no code release warrant caution.
Summary
The paper proposes online surrogate repair (OSR), a feedback regime for closed-loop AI-scientist workflows in which an agent searches mainly with inexpensive surrogate feedback while a sparse acquisition rule routes a small, scheduled budget of high-fidelity oracle queries to the agent's archived proposals; returned labels update a tabular foundation model's in-context table with no parameter training. Across 40 synthetic worlds the authors show global fit (NRMSE) and maximum regret are misaligned: coverage/uncertainty sampling improves fit but not regret, whereas Q90-UCB and expected improvement (EI) cut mean maximum-regret by about 0.41–0.42 versus roughly 0.05 for Random. In online synthetic experiments (200 episodes, a ReAct agent on DeepSeek-V4-Flash, TabPFN-3 surrogate), Online EI at Q=32 reaches 0.0552 mean regret versus 0.0984 for static EI, closing 76% of the gap to a 200-query oracle control. On MADE materials discovery (30 systems, MACE surrogate, Orb-v3 oracle), full-feedback controls need 6.36–7.23x more oracle queries (10.27x for the non-LLM Chemeleon+MLIP workflow) to match Online EI. Limitations: requires tabular design spaces and in-context surrogates; RL/test-time-training agents, other surrogate classes, and adaptive schedules untested; no code release stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Global surrogate fit (NRMSE) and optimizer-facing maximum regret are distinct, often anti-correlated repair objectives: in Table 1, Q90-UCB gains mean ΔR 0.4236 with mean ΔE −0.0002, while Random gains ΔE 0.0045 with ΔR only 0.0520.
- Online EI at Q=32 closes 76% of the gap between no repair (0.1432) and a 200-query oracle control (0.0278) on 20 synthetic worlds, cutting mean regret 44% versus static EI (0.0984 → 0.0552).
- On MADE, full-feedback controls require 6.36–7.23× more oracle queries (10.27× for Chemeleon+MLIP) to match Online EI's 32-query discovery counts, and these acceleration factors are lower bounds due to censoring.
- The benefit of online repair is not specific to in-context surrogates: Archive-BO with a GP surrogate achieves intermediate regret (0.0807, between Static EI 0.0984 and Online EI 0.0552).
- In the online setting, EI outperforms Q90-UCB (0.0552 vs 0.0600), reversing Q90-UCB's slight aggregate lead in the static study.
Strengths
Clear problem formulation: a well-defined third feedback regime
§3.2/Algorithm 1 cleanly separates an append-only proposal archive from an evenly spaced repair schedule, and Table 2 quantifies all three regimes (Q=0: 0.1432; Q=32: 0.0552; Q=200: 0.0278) at matched episode budgets.
Memorable fit-vs-regret dissociation with direct benchmark-policy implications
Table 1/Table 6: Global-UQ and Random achieve the best NRMSE gains (0.0046/0.0045) yet the worst regret reductions (0.0631/0.0520), while Q90-UCB gains ΔR 0.4236 with ΔE −0.0002 — a direct challenge to TabArena-style global-accuracy evaluation of feedback models.
Unusually thorough controls and ablations
Honest metric hygiene on MADE
Generality across proposal policies
Weaknesses
Statistical inference is thin for the central rankings
Possible prior-alignment circularity in the synthetic worlds
No code release and undocumented agent internals
Baseline scope gaps on the acquisition side
Regret metric definition shifts between sections
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is online surrogate repair (OSR)?
Online surrogate repair is a feedback regime in which an AI-scientist agent runs a long search using mostly cheap surrogate predictions, while a scheduled acquisition rule picks a few of the agent's archived proposals to receive expensive high-fidelity (oracle) evaluation. The returned labels are appended as rows to a tabular foundation model's context, so all later episodes are scored by an improved surrogate. No model parameters are retrained.
How many oracle queries does Online EI save?
On the MADE materials-discovery benchmark, controls that received high-fidelity feedback after every episode needed 7.23x (DeepSeek orchestrator) and 6.36x (GPT-5.6-Luna) more oracle queries to match Online EI's discovery count at 32 queries, and 10.27x for the non-LLM Chemeleon+MLIP workflow. These are lower bounds because some controls never caught up within their 300-query budget.
Why not just improve surrogate accuracy?
Because the paper shows the two objectives are misaligned: sampling rules that improve average predictive accuracy (Random, Global-UQ) barely reduce the maximum regret that actually determines which design an optimizer selects, whereas improvement-aware rules such as EI target exactly the decision-relevant regions. In Table 1, Q90-UCB cuts regret by 0.42 on average while slightly worsening global fit.
Does OSR require training or fine-tuning a model?
No. The surrogate is a pre-trained tabular foundation model (TabPFN-3 or TabICLv2). Repair simply appends newly measured (design, label) rows to the model's in-context table; the updated context changes future predictions without any parameter updates.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are represented only by captions and axis labels in the extraction; visual curve details could not be assessed.
- No section or appendix documents agent prompts or LLM sampling hyperparameters (Appendices A–F cover MADE metrics, world construction, sweeps, static results, costs, and Tab-AICL adaptation only).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.