WPBench: A Comprehensive Benchmark for Wind Power Forecasting
Yuhan Zhu, Jilin Hu, Xinying Cai, Yingshan Li, Li Ma, Xiangfei Qiu Linsen Li, Kai Zhang, Yao Fu, et al.
cs.LG · cs.AI
Quick verdict
WPBench unifies 26 wind datasets and 19 models from 5 families with shape and structure diagnostics; well-scoped, but no significance tests or visible code.
Summary
The paper introduces WPBench, a benchmark for wind power forecasting organized around wind-specific data structure. The authors curate 26 public datasets along two axes — turbine count and variable count — yielding four task types (single- vs multi-turbine, univariate vs multivariate), and evaluate 19 models from five families (statistical/ML, deep temporal, time-series foundation, deep spatio-temporal, spatio-temporal foundation) under one protocol: look-back 576, horizons 12–144, chronological 7:1:2 splits, Optuna tuning. Beyond point-wise errors, they add shape-aware metrics (DTW, nDTW, nDDTW, EDF), efficiency measurements, and structure-aware descriptors spanning temporal patterns, target–covariate dependencies, and cross-turbine spatial dependencies. Key findings: TOTO, FactoST-UTP, and TTM lead the normalized-accuracy leaderboard; linear models such as DLinear win nRMSE but lose curve fidelity; few-shot adaptation of foundation models often matches full-shot; spatial modeling brings sensitivity rather than guaranteed gains. Main limitations: no significance testing or seed variance; diagnostics rest on 20% quantile groups of roughly five datasets each; preprocessing rules stay qualitative; no repository link appears in the text; and outliers such as BigST's 0.722 nMAE on Chalmers go unexplained.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- WPBench is the first benchmark to explicitly cover all four wind power data-structure categories (1T1V, 1TMV, MT1V, MTMV), via 26 public datasets under a unified protocol.
- The best model by numerical accuracy is generally not the best by forecast-curve fidelity (e.g., DLinear vs FactoST-UTP on GEFCom2012: nMAE 0.190 vs 0.203 but nDTW 2.541 vs 2.172).
- Few-shot adaptation (10% of training data) of foundation models often matches or slightly exceeds full-shot adaptation, implying sample representativeness matters more than volume.
- Structure-aware diagnostics show temporal periodicity helps, covariate benefit requires stable dependencies, and spatial modeling yields sensitivity rather than guaranteed gains.
Strengths
Wind-specific structural organization
26 datasets sorted into 1T1V/1TMV/MT1V/MTMV (Table II) enables controlled contrasts that prior wind resources each cover only partially (Table I); the two-axis design directly ties task structure to the capabilities being tested.
Broadest cross-family comparison under one protocol
19 models from five families — including ST foundation models (FactoST-UTP/STA, OpenCity) and TSFMs (Timer, TTM, TOTO, SEMPO) — share identical splits, rolling windows, and Optuna tuning (Sec. IV-C/E), with scope-aware ranking for structurally inapplicable models (VAR absent on univariate tables).
Operationally aligned evaluation
Novel dynamic-structure diagnostics
Weaknesses
No uncertainty quantification
Undiagnosed outliers contaminate aggregates
Diagnostics rest on tiny groups
Qualitative curation and missing artifact link
Horizon aggregation mixes operational lead times
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
WPBench (arXiv 2609.24444, cs.LG) is a benchmark paper for wind power forecasting (WPF). The authors argue that general time-series benchmarks (TFB, fev-bench, GIFT-Eval) and wind-specific resources (GEFCom 2012/2014, SDWPF, OpenWPF) each cover only part of what WPF demands, and enumerate four gaps (Fig. 1): scenario coverage across turbine scale × variable composition, incomplete model-family coverage, point-error-only metrics, and temporal-only diagnostics. The response is a 26-dataset, 19-model benchmark organized into four structural task types (1T1V, 1TMV, MT1V, MTMV), with shape-aware metrics, efficiency measurements, and three-axis dataset characterization. The claimed primacy — 'first benchmark to explicitly cover all four structural categories' — is consistent with their Table I positioning but is difficult to verify exhaustively; it is plausible rather than demonstrated.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is WPBench?
WPBench is a benchmark for wind power forecasting that unifies 26 public datasets, 19 forecasting models from five families (statistical/ML, deep temporal, time-series foundation, deep spatio-temporal, spatio-temporal foundation), and multi-dimensional evaluation covering accuracy, forecast-curve shape, and computational efficiency, plus diagnostics of how data structure affects model behavior.
Which models perform best in WPBench?
On SeasonalNaive-normalized accuracy (nMAE), TOTO, FactoST-UTP, and TTM lead; on curve fidelity (nDTW), TOTO and FactoST-UTP lead. Simple linear models like DLinear top nRMSE but rank lower on shape metrics, showing that accuracy and curve fidelity select different winners.
Is WPBench's code and data publicly available?
All 26 datasets are individually public (Zenodo, Kaggle, competition releases), and the paper claims a reusable benchmark artifact supporting reproducible comparison, but the extracted text contains no repository link or DOI, so availability should be confirmed with the authors.
What are the four task types in WPBench?
Datasets are split by number of turbines and number of variables into single-turbine univariate (1T1V), single-turbine multivariate (1TMV), multi-turbine univariate (MT1V), and multi-turbine multivariate (MTMV), isolating temporal, covariate, and spatial modeling demands respectively.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are text-extracted only: the radar chart (Fig. 4), leaderboards (Fig. 5, numeric labels interleaved/garbled), and case-study curves (Fig. 10–12) cannot be visually verified.
- No appendix or supplementary material appears in the extraction; hyperparameter search spaces and per-model training configurations are absent.
- The metadata author list (9 names) differs from the full-text header (11 authors, including Weihao Jiang and Bin Yang).
- The paper asserts a benchmark artifact exists, but no URL/DOI is visible in the extracted text; availability could not be confirmed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.