Beyond a Universal Forecasting Selector: Demand-Conditioned Model Selection across Demand Patterns and Horizons
cs.LG
Quick verdict
No forecast selector wins everywhere across 9 demand datasets; CCG-AHSC/D lead on Smooth. Honest stats, but self-tuned rules and GRA-only evaluation limit it.
Summary
This study asks whether the model-selection mechanism itself should be treated as context-dependent in demand forecasting. Five selectors are compared on identical ground: single-metric RMSSE, rank-aggregating ERA, benchmark-relative OWA, and the authors' class-conditioned pipelines CCG-AHSC and CCG-AHSCD, each choosing among 24 optimized forecasting models (statistical, ML, and deep learning) across nine public datasets totalling 5,333 series classified as Smooth, Intermittent, Erratic, or Lumpy, three train-test splits (91:9, 80:20, 70:30), and horizons 1-12. Evaluation is ex post, using Global Relative Accuracy (GRA), a volumetric coherence measure, with Friedman tests, Holm-corrected Wilcoxon pairwise tests, and an ex post best-model reference as upper bound. No selector dominates: the CCG pipelines lead every Smooth configuration and two of three Erratic ones (statistically significant), while ERA and OWA lead the Lumpy and Intermittent settings, where most differences are not significant; preferred selectors also shift with training-data size and horizon. Main limitations: CCG weights and retention counts were fixed from preliminary analyses with no reported dataset separation or sensitivity analysis; the sole evaluation metric (GRA) also feeds CCG's screening features, creating possible circularity; demand classes are heavily confounded with datasets; no code release; no effect sizes or numeric selector-versus-best-model gaps.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No single model-selection mechanism consistently outperforms the others across demand classes, training-data regimes, and horizons (Table 3).
- CCG-AHSC and CCG-AHSCD are the strongest selectors for Smooth demand in all three partitions and for two of three Erratic configurations, with statistically significant differences.
- OWA and ERA lead several Intermittent and Lumpy configurations, and for Intermittent demand no selector difference is statistically significant.
- Selector suitability changes with historical data availability and horizon, so the selection mechanism should be treated as a context-dependent decision layer.
Strengths
Unified, large-scale comparison isolating selection logic
Five selectors act on the same pool of 24 models optimized under one criterion (HEF, §3.5.6-3.5.8), across 9 datasets (5,333 series), 3 partitions, horizons 1-12 — differences are attributable to the selection rule, not training budgets.
Honest inferential discipline
Shapiro-Wilk screening, Friedman global tests, Holm-corrected Wilcoxon pairs (§3.5.9); non-significant Intermittent results (p=0.1478/0.2720/0.2720) are explicitly reported as non-significant and no superiority is claimed (§4.2.2).
Ex post best-model upper bound reframes the problem
Above-average protocol reproducibility
Weaknesses
Self-referential tuning of the proposed selectors
Single-metric, potentially circular evaluation
Class-dataset confounding
Reporting gaps
Minor internal inconsistency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper reframes forecasting-model selection as a two-level decision problem: choosing a model, and choosing the selection mechanism itself. Five selectors - RMSSE (single metric), ERA (rank sum of MAE/RMSE/R2), OWA (benchmark-relative MASE+sMAPE vs Naive2), and the authors' CCG-AHSC / CCG-AHSCD pipelines - are compared on an identical pool of 24 optimized models across nine public datasets (5,333 series: 2,203 Smooth, 1,880 Intermittent, 229 Erratic, 1,021 Lumpy), three partitions (91:9, 80:20, 70:30), and horizons 1-12 (Section 3.6: 27 configurations). Central claims: no selector dominates; CCG variants lead Smooth everywhere and two of three Erratic configurations; OWA/ERA lead several Intermittent/Lumpy settings; suitability shifts with data availability and horizon. The claims are calibrated to the evidence - arguably the paper's most disciplined feature.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Global Relative Accuracy (GRA) in this paper?
GRA = 1 - |sum of forecasts - sum of actuals| / max(sum of actuals, 1) over the evaluation horizon. It measures how closely total forecasted demand matches total realized demand (volumetric coherence), not pointwise accuracy. Values near 1 are good; negative values mean the aggregate forecast error exceeded realized demand. Zero-demand cases are excluded.
Which forecasting-model selector should I use for intermittent demand?
According to this study, none can be declared best for intermittent demand: OWA had the highest median GRA under the 91:9 and 80:20 splits and CCG-AHSCD under 70:30, but Friedman tests found no statistically significant differences among the five selectors for intermittent demand in any split, so the ranking differences are descriptive only.
Is code available for CCG-AHSC and CCG-AHSCD?
No code repository is mentioned. However, the paper provides full pseudocode (Algorithms 1-6), the class-specific metric configurations (Table 2), CCG weights and retention counts (Section 3.3.4), and hyperparameter search spaces (Tables A1-A2), and states that all nine datasets are publicly available, so an independent reimplementation is possible.
Did any selector win everywhere in the experiments?
No. The class-conditioned CCG-AHSC/CCG-AHSCD pipelines led all three Smooth configurations and two of three Erratic configurations; ERA led Erratic and Lumpy under the 91:9 split; OWA led two Intermittent and two Lumpy configurations. The authors' headline conclusion is precisely that no universal selector exists.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 are image content; only captions and the in-text/Table 1-3 numbers were extractable, so per-horizon judgments rely on the authors' prose rather than the plotted curves.
- The provided arXiv metadata lists Adolfo González as sole author, while the paper text credits Víctor Parada as co-author (supervision and manuscript review); this review reflects the paper text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.