Agentic Search Spaces for Tabular Machine Learning
Renat Sergazinov, Artem Chistyakov, Sergey Pankevich, Artem Babenko
cs.LG
Quick verdict
Claims LLM-designed HPO search spaces lift tabular models ~0.6–2%; well-controlled, honest study, but gains are small and TabICLv2 comparison is confounded.
Summary
The paper asks whether LLM coding agents can design HPO search spaces for tabular ML that beat those shipped by model authors. Five families (MLP†, TabM†, RealMLP, LightGBM, TabICLv2) are decomposed into preprocessing, embedding, architecture, training, and inference modules; an agent (Claude Code with Opus 4.8; ablated with Codex/GPT-5.5) writes candidate implementations per module, and Optuna's TPE jointly optimizes module choices plus default hyperparameters under matched budgets (200 trials on small/medium datasets, 100 on large; 15 test seeds). Across 45 datasets, agentic spaces improve average relative performance by 0.5–0.8% for single models and 0.9% for DNN ensembles, about 2.0% on small-to-medium regression; meta-tuned RealMLP shows no gain. On TabArena-Lite, Elo improves for four of five families (up to +125 for MLP†) and the agentic TabICLv2 and RealMLP ensembles surpass AutoGluon best_quality. Strengths include a negative end-to-end autoresearch baseline (Table 3), stability checks over five space regenerations (Table 7), and pool-size/budget ablations. Limitations: small aggregate effects without significance tests on aggregates, a confounded TabICLv2 comparison (its default is untuned), per-fit compute up to 2.4× higher, and evaluation on only the first TabArena outer CV split.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Agent-generated module candidate sets, jointly optimized with default hyperparameters by classical HPO, improve nearly every model family over base author-provided spaces across 45 datasets (avg +0.6%, +2.0% on small-to-medium regression) at the same trial and ensembling budgets.
- The gains are robust to which space the agent produces: five independent re-generations per agent (Claude Opus 4.8 and Codex GPT-5.5) all beat the base MLP† space on all four aggregate metrics (Table 7).
- On the official TabArena-Lite protocol, agentic spaces raise T+E Elo for four of five families (up to +125 for MLP†), and the agentic TabICLv2 and RealMLP ensembles surpass AutoGluon 1.4 best_quality (1535).
- End-to-end agentic research loops (autoresearch) underperform both classical HPO (−0.1%) and the proposed method (+1.3% on four datasets at 50 matched trials), because they explore poorly, do not amortize cost, and require data access.
- The benefit comes from dataset-specific composition of a diverse candidate basis rather than any single winning module: winning combinations are pairwise distinct across datasets and performance grows monotonically with pool size (§5.1, §5.3).
Strengths
Matched-budget, multi-metric evaluation protocol
Base and agentic spaces share the same Optuna univariate TPE sampler (20 random startup trials), budgets (200 trials small/medium, 100 large; §3.1, Appendix B.5), 15 test seeds, and four aggregate metrics (mean rank, relative improvement, normalized score, Elo; Appendix B.3); the base space is a literal subset of the agentic space.
Honest negative baseline for end-to-end agents
§3.2 and Tables 3/16: the autoresearch loop (Karpathy) run 50 iterations on 4 datasets lands at −0.1% vs classical HPO at $20–50/dataset, while the proposed method reaches +1.3% at matched 50-trial budget; the failure is dissected into three concrete causes (narrow in-context exploration, inefficient search, no cost amortization).
Robustness checks against cherry-picking and space-size artifacts
Cost and contamination accounting
External validity on an official benchmark
Weaknesses
Small aggregate effects with no inferential statistics
Confounded TabICLv2 comparison
Missing expert-expanded-space baseline
'No extra cost' claim holds only for trial counts
TabArena evidence rests on one outer split with partial imputation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (Yandex/HSE; arXiv 2609.16309) proposes LLM coding agents as search-space designers for tabular ML. Each of five model families (MLP†, TabM†, RealMLP, LightGBM, TabICLv2) is split into five hand-chosen modules — preprocessing, embedding, architecture, training, inference (two axes for LightGBM/TabICLv2, whose learners are frozen) — and an agent (Claude Code with Opus 4.8 at max effort; ablated with Codex/GPT-5.5) writes up to 64 candidate implementations per task type without ever seeing target data. A classical Optuna TPE then jointly optimizes module choices plus each model's default hyperparameters per dataset. Central claims: (1) consistent gains over base author spaces across 45 datasets — 0.5% for DNNs, 0.8% for LightGBM, 0.5% for TabICLv2 single-model, 0.9% for DNN ensembles, ~2.0% on small-to-medium regression; (2) gains at matched tuning and ensembling budgets; (3) transfer to TabArena-Lite Elo for four of five families (up to +125 for MLP†), with agentic TabICLv2 and RealMLP tuned+ensembled above AutoGluon 1.4 best_quality (1535).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper 'Agentic Search Spaces for Tabular Machine Learning' actually do?
It uses an LLM coding agent (Claude Code with Opus 4.8) to write alternative code implementations for the modules of established tabular models — preprocessing, feature embeddings, architecture, training, and inference — and then lets a standard HPO algorithm (Optuna TPE) jointly pick modules and hyperparameters per dataset. On 45 datasets this beats the models' default HPO spaces by about 0.6% on average (up to 2% on small-to-medium regression) at the same tuning budget, and the generated spaces are released publicly.
Do LLM agents beat classical HPO for tabular ML?
Not when run end to end, according to this paper. An autoresearch-style loop that lets the agent edit the model code per dataset was 0.1% worse than classical HPO and cost $20–50 per dataset. The winning recipe keeps classical HPO for exploration and uses the agent only once per model (~$10) to enlarge the search space, without ever showing it the target data.
Can agentic search spaces beat AutoGluon?
Individual agentic single models do not beat AutoGluon's full multi-model system on the authors' own 45-dataset benchmark. On the TabArena-Lite benchmark, however, the tuned-plus-ensembled agentic TabICLv2 and RealMLP variants score above AutoGluon 1.4's best_quality preset (1535 Elo), though they remain behind AutoGluon 1.5 extreme_quality (1669), which also includes foundation models.
Is the code and the generated search spaces available?
Yes. The paper states that code, prompts, agent harnesses, and all five generated search spaces are released at github.com/yandex-research/agentic-hpset. Generation is dataset-independent — the agent never sees target data — so the spaces can be reused on new datasets with any standard HPO tool.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–5 appear only as garbled axis labels in the extraction; fANOVA magnitudes (Figure 3), Elo confidence intervals (Figure 5), and exact Figure 4 cell values could not be fully verified.
- Several segments were omitted by the text extractor (part of Section 4.1, Appendix D.1, parts of Tables 23 and 25); some aggregate numbers were cross-checked between the abstract, intro, and surviving appendix text.
- Agent model names and 2026-dated citations (Opus 4.8, GPT-5.5, TabICLv2) are taken as stated in the text and could not be cross-verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.