Are Gradient Boosting Models Suitable for Intermittent Demand Forecasting?
Vladislav Kislinskii, Mazhar Hameed
cs.LG
Quick verdict
GBMs alone underperform on intermittent demand but IMAPA+CatBoost gains up to 9.7% RMSSE; plausible, yet single-dataset with no significance tests.
Summary
This empirical study benchmarks eight forecasting approaches for intermittent demand on the Royal Air Force (RAF) dataset — 5,000 spare-part SKUs with monthly demand from January 1996 to December 2002 — across 3-, 6-, and 12-month horizons. Contenders span a classical method (ETS), specialized intermittent-demand techniques (SBA, TSB, ADIDA, IMAPA), gradient boosting (XGBoost, CatBoost), and a hybrid combining IMAPA's size forecasts with a CatBoost occurrence classifier. Evaluation uses RMSSE and MAE, each in unrounded and rounded (integer) form, reflecting indivisible retail units. Among individual models, specialized methods win: TSB and IMAPA lead RMSSE at short horizons, XGBoost ranks last everywhere, and CatBoost is best on MAE but weak on RMSSE due to occasional large errors. The hybrid improves on IMAPA by up to 9.74% RMSSE (3-month horizon) and 24.92% MAE, though the RMSSE gain decays to 1.32% at 12 months. Main limitations: despite the abstract's claim of evaluation across multiple datasets, only RAF is used; rankings lack significance tests and confidence intervals; the boosting models receive feature engineering and a 12-month validation set that univariate baselines lack; and only point-accuracy metrics are reported, with no inventory-relevant evaluation. Code is publicly available on GitHub.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Specialized methods (TSB, ADIDA, IMAPA) are the strongest individual models for intermittent demand on the RAF dataset across all tested horizons.
- Gradient boosting models underperform on their own: XGBoost ranks last on both metrics at every horizon, and CatBoost, while best on MAE, is among the worst on RMSSE.
- A hybrid combining IMAPA size forecasts with a CatBoost occurrence classifier improves accuracy over IMAPA alone — up to 9.74% RMSSE and 24.92% MAE (float, 3-month horizon).
- Model rankings depend substantially on metric choice and rounding, with differences averaging around 5% and reaching 18%.
Strengths
Validation-set design accounts for demand scarcity
Section 2 argues that a test-sized validation split (3.93–15.7% of data per Table 1) starves intermittent series of positive-demand examples, so GBMs get a fixed 12-month validation window and are retrained on train+validation — a thoughtful, rarely articulated choice.
Dual-metric, dual-rounding evaluation surfaces practitioner trade-offs
Table 3 reports RMSSE and MAE in both float and integer form, exposing the CatBoost paradox (best MAE 2.315 vs near-worst RMSSE 0.71 at horizon 3) and the SBA flip (worst MAE, mid-range RMSSE).
Honest self-limitation
Public artifacts on a standard benchmark
Multi-horizon design reveals where the hybrid helps
Weaknesses
Abstract overstates scope
No uncertainty quantification
Asymmetric baselines confound the headline claim
Hybrid underspecified and classifier unaudited
Point accuracy only, despite inventory motivation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether gradient boosting (GBM) is suitable for intermittent demand forecasting, benchmarking eight methods on the RAF dataset (Teunter & Duncan, 2009): 5,000 spare-part SKUs, monthly demand, Jan 1996-Dec 2002. Contenders: ETS (classical); SBA, TSB, ADIDA, IMAPA (specialized); XGBoost, CatBoost (GBM); and a hybrid that keeps IMAPA's size forecast but replaces its occurrence model with a CatBoost classifier. Evaluation covers 3-, 6-, and 12-month horizons using RMSSE and MAE, each in unrounded (float) and rounded (integer) form (Table 3). Central claims: (1) specialized methods are the strongest individual models; (2) GBMs alone underperform; (3) the hybrid improves accuracy by 'up to 10%'; (4) rankings are metric- and rounding-sensitive (~5% average shifts, up to 18%).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Are gradient boosting models suitable for intermittent demand forecasting?
According to this paper, not on their own. On the RAF spare-parts benchmark, XGBoost ranked last on both error metrics at all horizons, and CatBoost led on MAE but was among the worst on RMSSE. However, a simple hybrid pairing IMAPA's forecasts with a CatBoost demand-occurrence classifier did beat the best individual method.
How much does the IMAPA + CatBoost hybrid improve accuracy?
On the RAF dataset it reduced RMSSE by up to 9.74% and MAE by up to 24.92% versus IMAPA alone (unrounded predictions, 3-month horizon). The RMSSE gain shrank with horizon — 2.99% at 6 months and 1.32% at 12 months — while MAE gains stayed near 19%.
What is the RAF dataset?
A standard intermittent-demand benchmark introduced by Teunter and Duncan (2009): monthly demand records for 5,000 spare parts used by the UK Royal Air Force from January 1996 to December 2002.
Is the code for this paper available?
Yes. The authors state that all artifacts are publicly available at github.com/vkislinskii/intermittent_demand_forecasting_paper, built on Python's statsforecast, CatBoost and XGBoost libraries.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (hexbin density plot of ln(p) and v) cannot be assessed from the extracted text; the 'relatively homogeneous' dataset characterization is taken on trust.
- Aggregation of metrics across 5,000 series is not described; per-series distributions are unavailable, so dispersion around point estimates is unknown.
- The Conclusion's '~5% average, up to 18%' ranking-shift figures are not directly traceable to the displayed tables.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.