Artificial Neural Networks as Surrogate Models in Black Box Optimization
Md Khadimul Islam Zim, Martin Holeňa
cs.NE · cs.AI
Quick verdict
Proposes AFN-CMA-ES, an MLP-ensemble surrogate for CMA-ES; no significant win vs any baseline, 10D multimodal failures—honest but largely negative results.
Summary
The paper proposes AFN-CMA-ES, a surrogate-assisted variant of CMA-ES in which a bootstrapped ensemble of five MLPs (64-32 ReLU architecture) replaces most true objective-function evaluations. The surrogate is trained on rank-normalized targets from a local archive of at most 250 samples centered on the current CMA-ES mean, with a combined NLL+MSE loss, anchored regularization toward initial weights, and warm-started retraining after each true-evaluated generation. The number of subsequent surrogate-only generations ('model life') is chosen adaptively from Kendall's tau between previous surrogate predictions and new true values (thresholds 0.45/0.60/0.75). Evaluated on all 24 noiseless COCO/BBOB functions in 2D-10D with a 250d budget and 15 instances, against standard CMA-ES and the LQ, DTS, and LMM variants from the public COCO archive, the results are largely negative: Wilcoxon rank-sum tests (p<0.05, Holm-corrected) show no statistically significant advantage over any baseline on any function-dimension combination; LQ-CMA-ES is significantly superior on separable functions, and AFN-CMA-ES exhausts its budget on many multimodal functions at 10D (0/15 success on f3, f4, f15, f19-f24). The authors candidly attribute the failures to archive sparsity, MLP limits on multimodal landscapes, and bootstrap overlap. Main limitations: the abstract overstates competitiveness, the fixed 64-32 architecture contradicts a 'dimension-dependent' statement, and hyperparameters and ablations are underdocumented.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- AFN-CMA-ES integrates a bootstrapped MLP-ensemble surrogate with generation-based evolution control and a Kendall-tau-adaptive model life within CMA-ES, so most candidate evaluations run on cheap surrogate rankings.
- AFN-CMA-ES performs comparably to standard CMA-ES only on some unimodal, well-conditioned BBOB functions at low dimensions, which the authors themselves read as failure of the intended mechanism (Section 5.1).
- Across the full COCO/BBOB suite, AFN-CMA-ES achieves no statistically significant improvement over CMA-ES, LQ-CMA-ES, DTS-CMA-ES, or LMM-CMA-ES on any function-dimension combination (Wilcoxon, p<0.05, Holm-corrected; Section 5.3).
- The failures on multimodal functions at d>=5 stem from local-archive sparsity, limited MLP capacity, and bootstrap overlap, motivating future adaptive switching between global and local surrogate models (Sections 5.2-5.3, 6).
Strengths
Transparent reporting of a null result
Section 5.3 states explicitly that AFN-CMA-ES does not achieve statistically significant improvements over any baseline on any function-dimension combination, and Section 5.1 even interprets parity with plain CMA-ES as the method 'not behaving in the desired way' — rare candor for a paper proposing a new algorithm.
Rigorous standard benchmarking protocol
Full COCO/BBOB protocol (Section 4): 24 noiseless functions, d in {2,3,5,10}, budget B=250d, 15 instances, ERT ratios vs the best-2009 archive reference, bootstrapped 10-90% dispersion in Table 1, and Wilcoxon rank-sum tests with Holm correction over 24 functions (Sections 4.5, 5.3).
Method specified at reimplementation depth
Diagnostic analysis of failure modes
Weaknesses
Abstract overclaims relative to the evidence
Internal inconsistency in the architecture description
Underdocumented experimental configuration
No ablation studies
Results for 2D/3D/5D largely narrative-only
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (footnoted as accepted at ECML PKDD 2026) proposes AFN-CMA-ES, a surrogate-assisted CMA-ES in which a bootstrapped ensemble of five MLPs (64-32 ReLU, Eq. 1 loss combining NLL and MSE, anchored regularization Eq. 2) replaces most true objective-function evaluations. Framed as a small-data learning problem (Section 1), the aims (Section 1.1) are to cut costly evaluations while staying near-optimal and to characterize dependence on benchmark-function character. The abstract's headline claim of 'competitive performance compared to state-of-the-art surrogate-assisted BBO methods' is not supported by the paper's own statistics, which find no significant improvement over any baseline (Section 5.3). The body is markedly more honest than the abstract.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AFN-CMA-ES?
A surrogate-assisted version of the CMA-ES evolutionary optimizer in which an ensemble of five small neural networks (MLPs) predicts objective values and their uncertainty, allowing several consecutive generations to run on cheap predictions instead of expensive true function evaluations; the number of such surrogate-only generations adapts to the surrogate's ranking accuracy measured by Kendall's tau.
Does AFN-CMA-ES outperform standard CMA-ES?
No. The paper's own statistical analysis (Wilcoxon rank-sum, p<0.05 with Holm correction) finds no significant improvement over any of the four compared baselines on any of the 24 BBOB functions in 2D-10D. It only matches plain CMA-ES on some easy unimodal functions at low dimensions, and fails badly on many multimodal functions in 10D (0/15 successes on f3, f4, f15, f19-f24).
What benchmarks and baselines were used?
The COCO/BBOB noiseless suite (24 functions) at dimensions 2, 3, 5, and 10, with a budget of 250 times the dimension evaluations and 15 independent instances per setting. Baselines (CMA-ES, LQ-CMA-ES, DTS-CMA-ES, LMM-CMA-ES) were taken from the public COCO result archive rather than re-run.
Is code available for this paper?
The paper provides no code or repository link. However, the method is fully specified in Algorithm 1 and Equations 1-5, the benchmark is the public cocoex platform, and baseline data come from the public COCO archive, so an independent reimplementation is feasible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are available only as extracted axis labels and captions; exact ECDF curve positions and RMSE trajectories could not be analyzed numerically.
- Per-function results for 2D/3D/5D exist only in figures and brief narrative; Table 1 covers 10D exclusively, so lower-dimension judgments rest on the authors' text.
- Math extraction is imperfect in places (Eq. 1 notation, the 'up-arrow 1' cell in Table 1 for f17); these were treated as formatting artifacts rather than substantive content.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.