How Faithful Is Attribution for Sales Forecasting? A Counterfactual Study
cs.LG
Quick verdict
Post-hoc counterfactual attribution for a WaveNet sales forecaster, significance-tested faithfulness; candid but small-n evidence and a placeholder code link.
Summary
The paper attaches a post-hoc, architecture-agnostic counterfactual interpretability layer to a WaveNet-style multi-series sales forecaster trained on the full Corporación Favorita grocery panel (174,685 series, 1,688 days). Starting from a fully baselined input, groups of input positions (seasonal pattern, recent trend, promotion) are revealed cumulatively in a fixed order; each group's contribution is the measured change in prediction, so contributions telescope to an exact sum. The authors argue this avoids the near-zero group shares they observed when additive SHAP attributions are summed over small or canceling groups. Faithfulness is tested with a deletion/insertion protocol on 30 randomly sampled series: attribution-ordered perturbation beats random ordering (deletion gap 0.217, p<0.001; insertion gap 0.271, p<0.01), stable across five SHAP background seeds. Application analyses show promotion reliance is heterogeneous (median day-type attribution ratio 1.02 among 40 series with 15–85% promotion frequency; ~20% exceed ratio 2) and that the model reproduces the weekly-cycle shape (r=0.78 over 300 series) while damping amplitude. Forecast accuracy is deliberately secondary (NWRMSLE 0.6137→0.6102 with the promotion covariate). Main limitations: very small evaluation samples relative to the panel, unspecified random-baseline variance and gap units, a single dataset and architecture, and a placeholder GitHub URL in the code footnote that undermines the stated open-source release.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Sequential counterfactual attribution decomposes multi-series forecasts into contributions that sum exactly to the predicted value (Section 5).
- Attribution orderings beat random orderings on deletion (gap 0.217, p<0.001) and insertion (gap 0.271, p<0.01) tests, stable across five SHAP background seeds (Section 6).
- Promotion reliance is heterogeneous: median attribution ratio 1.02 across 40 series with 15–85% promotion frequency, with about 20% of series above 2 (Section 7.1).
- The model captures the weekly-cycle shape (r=0.78 over 300 series) while systematically damping its amplitude (Section 7.2).
- Additive SHAP-style group attribution collapses for small or canceling groups, motivating the counterfactual construction (Section 5).
Strengths
Faithfulness testing is rare in forecasting and is done properly here
Section 6 adapts the Petsiuk et al. (2018) deletion/insertion protocol with one-sample t-tests (deletion t=4.95, p=2.9e-5; insertion t=3.29, p=2.7e-3, n=30) and reports five-seed robustness ranges (deletion 0.216–0.263; insertion 0.269–0.284), rather than showing attributions untested.
Exact-sum decomposition with a concrete motivating failure mode
The telescoping construction (Section 5) guarantees baseline prediction plus contributions equals the forecast, and is motivated by an observed SHAP artifact: an always-promoted item's promotion bucket received ~0 attributed share.
Unusually candid limitation reporting
Bias-aware evaluation sampling
Reproduction-grade model specification
Weaknesses
Placeholder code URL left in the manuscript
Very small evaluation samples relative to the panel
Statistical protocol under-specified
'Architecture-agnostic' is asserted, never demonstrated
The SHAP-collapse motivation is anecdotal
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper pursues a narrow, well-scoped question: can a post-hoc counterfactual attribution layer be attached to an already-trained multi-series deep forecaster, and are the resulting explanations *faithful* to the model rather than merely plausible? The artifact is a sequential masking construction (Section 5): from a fully baselined input window, groups of positions (seasonal pattern, recent trend, promotion) are revealed cumulatively in a fixed order; each group's contribution is the measured prediction change, and telescoping guarantees contributions sum exactly to the forecast. The authors motivate this against additive SHAP-style allocation, reporting that per-position SHAP summed within a group collapses to ~0 for an always-promoted item (Section 5) — a concrete failure mode, though asserted from observation rather than quantified in the paper. Claims are deliberately modest: no accuracy advance (NWRMSLE 0.6137 → 0.6102, reported 'for completeness'), but faithful, deployable explanation plus an honest account of where explanation is informative (Sections 7–8).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper's counterfactual attribution method do?
It explains a single forecast from a multi-series deep forecaster by starting from an input where every channel is replaced by a neutral baseline value, then revealing groups of true input positions one at a time in a fixed order. Each group's contribution is the actual change in the model's prediction when that group appears, so all contributions sum exactly to the final forecast.
Is the attribution faithful to what the model actually computes?
The authors test this with deletion and insertion protocols on 30 randomly sampled series: masking or revealing inputs in attribution order changes the forecast significantly faster than random order (deletion gap 0.217, p<0.001; insertion gap 0.271, p<0.01), and the result holds across five background-sampling seeds. Per-series effects are heterogeneous, so individual series can deviate substantially from the average.
Does the model rely on promotions to forecast sales?
Only for a minority of series. Among 40 series with promotion frequency between 15% and 85% of days, the median ratio of mean absolute attribution on promotion days versus non-promotion days is 1.02 (essentially no difference), while roughly 20% of series show a ratio above 2.
Is the code and data for this paper available?
The paper states the method is implemented in an open-source Python package (wavexplain, installable via pip as version 0.1.0) and uses the public Kaggle Corporación Favorita dataset. However, the GitHub URL printed in the preprint is an unfilled placeholder ('YOUR-USERNAME'), so the repository link cannot currently be verified from the paper itself.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 appear only as captions in the extracted text; the actual curve shapes (e.g., Figure 2 deletion/insertion curves) and bar values could not be visually inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.