Accuracy Is Not Service: A Decision-Aware Benchmark for Intermittent-Demand Forecasting
Joo Ern Chin, Shih-Fen Cheng, Aldy Gunawan
cs.LG
Quick verdict
Claims accuracy rank inverts against order-service on intermittent demand (ρ=−0.555); well-executed, though the inversion weakens under per-method tuning.
Summary
The paper benchmarks 38 forecasting methods—classical, intermittent-demand specialists, ML, deep learning, and pretrained foundation models—on a contract-logistics spare-parts panel (4,021 materials, 70% zero months) plus public RAF and M5 panels. Under a common order-up-to policy replaying 20,330 real multi-item orders, the Spearman correlation between MASE rank and complete-order fill-rate rank is −0.555 (95% CI [−0.78, −0.26]) on the industrial panel, and only over-forecasting methods (Croston, SBA, ADIDA) meet an 80-of-80 service target, at 1.2–1.5× Naive's inventory. Tracing bias to Chronos-2's instance normalization, which centers on a zero-dominated mean, the authors propose a training-free non-zero-conditional re-centering that raises the per-material fill proxy from 0.775 to 0.920 (+14.5pp) and order-level fill from 0.54 to 0.63, at worse MASE (0.831→0.948). Findings replicate on RAF (+17.1pp) and on RUF, a fidelity-certified synthetic 5,000-material surrogate released under CC-BY-4.0 (inversion ρ=−0.47). Key limitations: the inversion attenuates to ρ=−0.278 (p=0.09) when each method gets its own cost-optimal safety factor; core evidence rests on one confidential monthly domain, with the public surrogate generator-dependent; and the MASE convention is non-standard.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On the industrial spare-parts panel, forecast-accuracy rank and complete-order fill-rate rank correlate negatively (ρ=−0.555, 95% CI [−0.78, −0.26]) across 38 methods scored on 20,330 real multi-item orders under a common order-up-to policy.
- Service is associated with the direction of cumulative forecast bias — over-prediction during zero-demand periods — more than with point accuracy.
- A training-free re-centering of Chronos-2's instance normalization on non-zero-conditional statistics lifts the per-material fill proxy from 0.775 to 0.920 and complete-order fill from 0.54 to 0.63, at worse MASE (0.831→0.948).
- The findings reproduce on the released fidelity-certified synthetic RUF panel (inversion ρ=−0.47) and the correction replicates on public RAF (+17.1pp).
- The inversion is regime-specific: it weakens on RAF (−0.173, n.s.), reverses on low-intermittency M5 (+0.359), and attenuates under per-method cost-optimal policy tuning (−0.278, p=0.09).
Strengths
Unusually candid evaluation protocol
The audited no-fallback rule (Section III) counts method failures instead of substituting baselines; the non-standard MASE (Eq. 2) is explicitly flagged; the Nemenyi CD=10.90 is reported even though it dissolves the accuracy leaderboard; and the per-method z-tuning attenuation (ρ=−0.278, p=0.09) is disclosed in Section IV-A and Table VII rather than buried.
Mechanism plus actionable fix, not just a correlation
The inversion is decomposed via the zero-period MAE term (Eq. 5: 53–54% of Croston/SBA error vs ≈26% for Naive) and signed drift (Croston +8.9 vs TimesFM −8.3), then converted into a training-free Chronos-2 re-centering (Eq. 7) with quantified gains: fill proxy 0.775→0.920 (+14.5pp) on industrial, +17.1pp on RAF (0.607→0.778), COFR 0.54→0.63.
Regime-contrast design supports a falsifiable claim
Reproducibility infrastructure for private-data findings
Weaknesses
Headline correlation is policy-constructed
Core evidence is unverifiable in principle
Model-family mismatch in the celebrated correction
Non-standard accuracy metric complicates external comparability
Inventory cost of the fix is under-reported at order level
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper benchmarks 38 forecasting methods across eight model classes on intermittent spare-parts demand, arguing that the standard selection pipeline (accuracy via MASE/sMAPE → base-stock level → presumed service) fails under heavy intermittency. Central claims: (1) accuracy rank and complete-order fill-rate (COFR) rank correlate at ρ=−0.555 on the confidential industrial panel (20,330 real multi-item orders, 3,828 materials, 41 platforms); (2) bias direction, not point accuracy, explains service; (3) a training-free re-centering of Chronos-2's instance normalization lifts the per-material fill proxy 0.775→0.920 and COFR 0.54→0.63; (4) the findings reproduce on the released RUF synthetic panel (ρ=−0.47). The body scopes these claims carefully — regime-specific, common-policy — but the title and abstract present the inversion without its most important qualification.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why can the most accurate demand forecast deliver worse spare-parts service?
Under the order-up-to policy studied, the most accurate models (WMA, TimesFM, Chronos-Bolt) systematically under-predict, which sets lower stock targets. On an industrial spare-parts panel with 70% zero-demand months, rank by accuracy correlated negatively (ρ=−0.555) with rank by complete-order fill rate across 38 methods replaying 20,330 real orders, while methods with positive cumulative bias (Croston, SBA, ADIDA) held 1.2–1.5× more inventory and met the service target. The effect weakens when each method tunes its own safety factor.
What is the Chronos-2 normalization fix and how much does it help?
Chronos-2 normalizes each input series by its overall mean, which is dominated by zeros in intermittent demand. Re-centering on the mean and standard deviation of positive (non-zero) observations — with no retraining — raised the per-material fill proxy from 77.5% to 92.0% on the industrial panel and by +17.1pp on RAF, and lifted complete-order fill from 0.54 to 0.63. The cost is worse point accuracy (MASE 0.831→0.948) and higher stock targets.
Is the paper's dataset public?
No — the industrial panel is confidential. The authors release RUF, a fidelity-certified synthetic 5,000-material surrogate (CC-BY-4.0), plus generators, evaluation code, and public-panel leaderboards at github.com/sfcheng-research/icdm-2026-reproduction; the leaderboard, inversion direction, and correction reproduce on it (ρ=−0.47), while contractual magnitudes remain private-panel results.
Does the accuracy–service inversion hold for all kinds of demand?
No. It is strongest on the heavily intermittent industrial panel (order-level ρ=−0.555), weakens to −0.173 (not significant) on the sparser public RAF panel, and reverses to +0.359 on the less intermittent M5 retail panel. It also attenuates (ρ=−0.278, p=0.09) when each method is given its own cost-optimal safety factor, so the authors scope the claim to heavy intermittency under common policies.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures were extracted as scattered axis-label and caption text (e.g., Fig. 2's per-segment rank grid appears as a jumbled numeral block); plotted detail beyond the printed captions and numerals could not be inspected.
- The full 38-method accuracy roster is not in the paper itself — Table III shows 18 of 38 methods, with the remainder deferred to the release repository (paper-side, not an extraction loss).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.