SAGE-Loop: Reliable Closed-Loop LLM-Driven AutoML with Trial-and-Correction and Adaptive Ensembling
Junquan Gu, Shibo Cui, Xiangfeng Luo, Hang Yu
cs.LG
Quick verdict
LLM closed-loop AutoML with trial-and-correction and adaptive ensembling; gains look plausible but dataset accounting and statistical rigor lag the claims.
Summary
SAGE-Loop is an LLM-driven AutoML framework for tabular data that wraps pipeline generation in a closed loop. Its three components are: (1) feedbackable feature engineering, where an LLM proposes feature-engineering code and task-aware criteria (mutual information, F-test, variance) retain a top-N subset; (2) multi-round model synthesis, where executability checks, error logs, and validation scores trigger error-driven repair or performance-driven revision prompts; (3) adaptive ensembling — supervised tasks select among stacking, bagging, and voting with an LLM-synthesized level-2 learner, while unsupervised tasks fuse heterogeneous clusterings via a co-association matrix, spectral clustering, automatic k estimation, and filtering of degenerate outputs. On 20 public datasets, the paper reports best or second-best AUC/ACC on eight classification datasets (including 100.00% AUC/ACC on balance-scale), the lowest MAE/RMSE/RMSLE on all four regression datasets, and the best mean score in 17 of 20 clustering dataset-metric comparisons, plus 5/5 run completion on seven datasets in a failure-recovery analysis. Main limitations: inconsistent dataset accounting between setup and tables, no significance testing despite small margins, a 200-token response limit that conflicts with full-classifier generation, partially unspecified LLM configurations, uneven baseline tuning, and code not yet released.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 5
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A process-level trial-and-correction loop (error- and performance-driven prompt rewriting) lifts autonomous end-to-end completion to 5/5 on seven datasets where baselines fail without manual repair (Sec. 5.3, Fig. 3).
- Multi-round LLM model synthesis yields a diverse, valid, high-quality base-model pool that single-shot generation cannot (Sec. 4.2).
- Adaptive, validation-evidence-based selection among stacking/bagging/voting with an LLM-synthesized level-2 learner outperforms fixed ensemble templates (Sec. 4.3, Fig. 4).
- Consensus clustering via co-association matrix, spectral clustering, automatic k estimation, and degenerate filtering wins 17 of 20 dataset-metric comparisons (Table 3).
- Across 20 public datasets SAGE-Loop consistently improves performance and stability over traditional AutoML (AutoGluon, H2O, TPOT), tree ensembles, and LLM baselines (DS-Agent, CAAFE).
Strengths
Unified supervised + unsupervised closed loop
Adaptive ensemble selection (Sec. 4.3) and consensus clustering (Sec. 4.4) share one feedback cycle; unsupervised AutoML baselines are thin in the field, and SAGE-Loop beats Kmeans/GMM/Agglomerative/DS-Agent in 17/20 dataset-metric comparisons (Table 3, e.g., glass ARI 26.69 vs 20.70 for GMM).
Honest separation of autonomy from assisted performance
Sec. 5.3 evaluates completion on unmodified baselines (DS-Agent 0/5 on boston/california; CAAFE 3/5) separately from Tables 1-3, where baselines received manual repair — an unusually explicit disclosure that SAGE-Loop alone ran without human intervention.
Ablations localize gains to both modules
Practical deployment footprint and inspectable loop
Seed variance reported rather than hidden
Weaknesses
Benchmark accounting does not reconcile between setup and tables
Statistical support is rank-counting, not inference
Setup contradictions impair reproducibility
Uneven baseline treatment and missing strong baselines
Component novelty is modest
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SAGE-Loop (arXiv 2609.12455) targets two diagnosed pathologies of AutoML: the absence of process-level trial-and-correction (one-way pipelines terminate or bypass failures) and diversity that is generated but never used in an evidence-based manner. Claimed contributions: (i) multi-round LLM-guided model synthesis with execution-aware repair, (ii) a unified adaptive ensemble — stacking/bagging/voting with an LLM-synthesized level-2 learner for supervised tasks, co-association consensus for unsupervised ones, (iii) a prompt-execution closed-loop workflow. Headline empirical claims: best or second-best AUC/ACC on all eight reported classification datasets, lowest MAE/RMSE/RMSLE on all four regression datasets, best mean score in 17/20 clustering dataset-metric comparisons, and 5/5 autonomous completion on seven failure-recovery datasets.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SAGE-Loop?
SAGE-Loop is an LLM-driven AutoML framework for tabular data that closes the generation loop: an LLM writes feature-engineering and model code, failures are repaired automatically from error logs and validation scores, and ensemble strategy is chosen adaptively — stacking/bagging/voting with an LLM-synthesized level-2 learner for supervised tasks, and consensus clustering over a co-association matrix for unsupervised tasks.
How does SAGE-Loop compare with AutoGluon, H2O, or TPOT?
On the paper's 20 public datasets it reports best or second-best AUC/ACC on eight classification datasets (including 100.00% on balance-scale), the lowest MAE/RMSE/RMSLE on all four regression datasets, and the best mean clustering score in 17 of 20 dataset-metric comparisons — reported as mean±std over five seeds, but without significance testing, so small margins (e.g., 0.03 ACC on cd2) should be read cautiously.
Is the SAGE-Loop code available?
Not yet. The paper states that code, prompts, and reproduction scripts 'will be released upon publication'; no repository is linked in this preprint (arXiv 2609.12455). Four prompt templates and key hyperparameters (5 rounds, top-10 features, 10 candidates, 10 hyperparameter trials each) are described in the text.
Which LLM does SAGE-Loop use?
The paper says LLM components call the OpenAI API with temperature 0.5. Regression results are explicitly labeled GPT-3.5-turbo ('OURS(gpt3.5)'), but the model behind the classification and clustering tables is not clearly specified — a notable gap, since key baselines (DS-Agent) use gpt-4o.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3-7 appear only as captions in the extracted text; the plotted values behind completion counts (Fig. 3), stacking-variant comparisons (Fig. 4), and k-sensitivity curves (Fig. 5) cannot be checked from text, so figure-based judgments rest on captions and surrounding prose.
- Table cells were extracted as linear text; row/column alignment was reconstructed and cross-validated against ablation values (credit-g AUC 81.08±0.24 and concrete RMSE 4.14±0.33 match Table 4 'w all'), but a minor misalignment risk remains for dense rows.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.