Investigating Hyperparameter Optimization and Transferability for ES-HyperNEAT: A TPE Approach
Romain Claret, Michael O'Neill, Paul Cotofrei, Kilian Stoffel
cs.NE · cs.AI
Quick verdict
TPE beats random search for ES-HyperNEAT tuning on MNIST (29.0% best acc); honest bug validation, but no test set and best-of-runs headline vs prior 23.9%.
Summary
This GECCO '24 Companion paper applies Bayesian optimization — specifically the Tree-structured Parzen Estimator (TPE) via Optuna — to tune sixteen ES-HyperNEAT hyperparameters (over 3 billion combinations) on MNIST classification, then tests transferability to Boolean logic operations and Fashion-MNIST. A 2,013-trial TPE search (78 days, 20 generations per trial, 200-image batches) significantly outperformed 292-trial random search (mean accuracy 17.41% vs 11.41%; best 27.5% vs 20.5%). The best configuration, validated over 30 runs, reached 28.0% best accuracy; a post-hoc validation after a parallel-execution bug (which swapped configurations between concurrently running trials) found a configuration reaching 29.00% best / 20.95% mean, exceeding the prior HyperNEAT-with-LeNet-5 result of 23.90% with a smaller population and far fewer generations. Transferred hyperparameters significantly improved Fashion-MNIST (mean 20.00% vs 11.93% random) and three logic operations (XOR, OR, NAND), but not AND, NOR, or XNOR. Main limitations: accuracies are fitness on training batches with no held-out test set; the headline 29.00% is a best-of-runs statistic; roughly 25 t-tests lack multiple-comparison correction; absolute performance remains only modestly above the 10% chance level; and no code is released.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- TPE significantly outperforms random search when optimizing 16 ES-HyperNEAT hyperparameters on MNIST (TPE-Search mean 17.41% vs RandomSearch-292 11.41%, large effect sizes, Tables 2-3).
- The best TPE-found configuration achieves 29.00% best / 20.95% mean validation accuracy on MNIST, exceeding the prior 23.90% HyperNEAT result with a smaller population (≤100 vs 256) and far fewer generations (20 vs 2,500).
- MNIST-optimized hyperparameters transfer effectively to Fashion-MNIST (mean 20.00% vs 11.93% random search) but transfer significantly to only XOR, OR, and NAND among the six logic operations.
- A parallel-execution bug inflated random-search baselines for AND, XOR, NOR, and XNOR, yet corrected validation confirms TPE superiority: 72 of 199 validated configurations beat the best corrected random-search result of 22.00%.
Strengths
Candid bug disclosure and systematic validation
Section 5.3: upon discovering that parallel execution swapped configurations between concurrent trials, the authors halted, fixed, and ran a 15-day validation — 271 unique configurations sampled by a rank-plus-neighborhood scheme, 199 validated (≥3 trials each), 8 top configs x 30 trials — and quantified exactly which baselines the bug inflated (AND, XOR, NOR, XNOR random search, Table 10).
Above-average statistical reporting for a companion-track paper
Welch t-tests with Cohen's d across four tables (3, 6, 8, 10); a sanity check that 30 random trials suffice (RS-30 vs RS-292: t = 0.11, p = 0.912, Table 3); worst-case reporting of the Fashion-MNIST result where the transferred config loses on best accuracy (22.50% vs 23.50%, Table 7).
First systematic HPO of ES-HyperNEAT
Efficiency-relevant headline result
Cleanest transfer evidence on Fashion-MNIST
Weaknesses
No held-out test set anywhere
Best-of-runs headline compared to an aggregation-unknown prior number
No multiple-comparison control across ~25 tests
Near-chance absolute performance vs strong framing
Single optimizer, non-compute-matched baseline
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This GECCO '24 Companion paper (corrected arXiv v2, results unchanged per the footnote) asks whether Bayesian optimization can replace hand-tuning for ES-HyperNEAT, an indirect-encoding neuroevolution algorithm. The authors select 16 hyperparameters (Table 1) spanning NEAT core parameters (pop_size, connection/node mutation probabilities, initial connection type, activation defaults) and substrate parameters (initial/max depth, variance and division thresholds, max weight, iteration level), for a verifiable search space of ~3.2 billion combinations. Three claims structure the paper: (1) TPE (via Optuna) outperforms random search on MNIST; (2) the best found configuration reaches 29.00% validation accuracy on MNIST, surpassing the 23.90% of Verbancsics & Harguess [19] with a smaller population (≤100 vs 256) and far fewer generations (20 vs 2,500); (3) the configuration transfers to Fashion-MNIST but only partially to logic operations (XOR, OR, NAND significant; AND, NOR, XNOR not).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How accurate is ES-HyperNEAT on MNIST with TPE-optimized hyperparameters?
In this paper, the best validated configuration reaches 29.00% best-case and 20.95% mean accuracy over validation runs (chance is 10%). These figures are fitness measured on 200-image training batches, not a held-out test set. It still exceeds the earlier HyperNEAT result of 23.90% (obtained with a predefined LeNet-5 substrate, population 256, 2,500 generations) while using a population of at most 100 and only 20 generations.
Did the MNIST-tuned hyperparameters transfer to other tasks?
Yes for Fashion-MNIST: the transferred configuration reached 20.00% mean accuracy versus 11.93% for random search, a large effect. For the six Boolean logic operations, transfer was statistically significant only for XOR, OR, and NAND; AND, NOR, and XNOR showed no significant difference from random search.
What bug did the authors find and how did they handle it?
Their parallel execution code occasionally swapped hyperparameter configurations between concurrently running trials. They halted experiments, fixed the code, and ran a 15-day validation re-evaluating configurations from the original TPE search (199 of 271 sampled configurations validated). The bug significantly inflated random-search baselines for AND, XOR, NOR, and XNOR logic tasks, but did not materially affect the MNIST, OR, NAND, or Fashion-MNIST comparisons.
Is code or data available for this paper?
No code repository or experiment logs are referenced in the paper. The stack is public — Pureples, neat-python, Optuna, and the MNIST/Fashion-MNIST datasets — and the key hyperparameter configurations are tabulated, so reimplementation is possible, but exact reproduction of the 78-day, 13-machine search is not documented in reproducible form.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is available only as extracted text (axis labels and annotations), so visual trends in the optimization trajectory cannot be assessed.
- The expected-improvement formula in Section 2.2 shows l(x) and g(x) juxtaposed where the standard TPE formulation has a ratio; superscripts and fraction bars are lost in PDF extraction, so this may be an extraction artifact rather than an error in the paper.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.