Overlay\_dx - Automating forecasting evaluation
Long Ngo, Mohammed Amine Chamli, Jonathan Rivalan, Thomas Jaillon
cs.LG · cs.AI
Quick verdict
Proposes overlay_dx, an AUC score over accuracy-vs-tolerance curves for forecast evaluation; intuitive and open-source, but novelty and rigor are thin.
Summary
The paper introduces overlay_dx, an evaluation metric for time-series forecasting that combines visual and numerical assessment. The 'overlay' is the percentage of predicted values falling inside a tolerance band of width x around the actual series; sweeping the tolerance across decreasing thresholds yields an overlay curve, and the area under this curve, normalized by the maximum percentage, gives the overlay_dx score. The authors provide pseudocode, an open-source implementation (Smile-SA/overlay_dx on GitHub), and experiments with eight classical models (linear regression, Random Forest, XGBoost, LightGBM, k-NN, Extra Trees, Bagging, ARIMA) on three public datasets (Beijing air quality, ETT, Electricity Load Diagrams) plus a synthetic suite with controlled variations. On the synthetic data, overlay_dx correlates -0.62 to -0.5 with MAE/RMSE/MSE and only weakly (slightly positively) with MAPE/NRMSE. In Table 2, Extra Trees scores best on pollution data (overlay_dx 0.952), Random Forest on ETT (0.674), and ARIMA on ELD (0.847). Main limitations: no statistical significance testing, unstated threshold-grid hyperparameters, unexplained metric anomalies (ARIMA WMAPE 0.00; ELD MAPE ~1e15), and no positioning against existing tolerance-curve or interval-coverage metrics.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- overlay_dx unifies visual and numerical forecast evaluation: overlay curves over tolerance thresholds plus a normalized AUC score in [0,1].
- On synthetic data, overlay_dx correlates -0.62 to -0.5 with MAE/RMSE/MSE and only weakly (slightly positively) with MAPE/NRMSE, showing it captures complementary performance aspects (Fig. 2).
- Across three public datasets and eight classical models, overlay_dx discriminates models usefully: Extra Trees 0.952 (Pollution), Random Forest 0.674 (ETT), ARIMA 0.847 (ELD).
- The metric is robust to outliers and more interpretable for non-technical stakeholders than traditional error metrics (§1, §3).
Strengths
Dual visual + numerical design is genuinely usable
The overlay curve (Fig. 1b, Fig. 4) makes threshold-dependent behavior visible in one plot, while the normalized AUC (Alg. 2) collapses it to a single comparable score; §4.2's Fig. 4 shows this working across all three datasets.
Open-source implementation
§1 states the metric ships as an open-source project (github.com/Smile-SA/overlay_dx) with documentation, plotting utilities, and ML-framework integration, supporting adoption and partial reproducibility.
Sensible evaluation protocol
Transparent anomaly reporting
Weaknesses
No statistical rigor behind rankings
Novelty under-positioned
Metric hyperparameters unstated
Unexplained anomalies suggest pipeline issues
Robustness claims unsubstantiated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This 11-page industry paper (Smile SA, funded by EU NEPHELE grant 101070487) proposes **overlay_dx**, an evaluation metric for point forecasts of time series. The 'overlay' is the share of predictions whose absolute error is at most a tolerance x (Alg. 1); sweeping x over a grid of percentage thresholds relative to the target's range (x = pct/100 · (max−min)/2, Alg. 2) traces an 'overlay curve' whose normalized area under the curve is the overlay_dx score. The claimed contributions (§1) are: (1) a new visual metric, (2) a quantitative AUC-based score, (3) demonstrated effectiveness across scenarios, (4) a bridge between technical evaluation and stakeholder interpretability. Scope is strictly classical point-forecast evaluation — no probabilistic forecasts; multivariate extension is deferred to §5 future work.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is overlay_dx and how is it computed?
overlay_dx is an evaluation metric for time-series forecasting. For each tolerance width x (a fraction of the target value range), it computes the percentage of predictions whose absolute error is at most x. Sweeping x from large to small gives an 'overlay curve'; the area under this curve, normalized by the maximum percentage, is the overlay_dx score between 0 and 1 — higher means predictions track the actual series more tightly across tolerance levels.
Is the code and data for this paper publicly available?
Yes. The authors released an open-source implementation with documentation and plotting utilities at github.com/Smile-SA/overlay_dx. The experiments use three public datasets: Beijing Multi-Site Air-Quality (2013-2017), ETT (electricity transformer temperature, from the Informer paper), and Electricity Load Diagrams 2011-2014 (UCI repository).
Which forecasting model performed best according to overlay_dx?
It depends on the dataset (Table 2): Extra Trees scored highest on the Beijing air-pollution data (overlay_dx 0.952), Random Forest on ETT (0.674), and ARIMA on the Electricity Load Diagrams data (0.847). Notably, ARIMA leads on ELD despite Extra Trees having the better RMSE there (5.632 vs 6.079), which the paper does not analyze in depth.
Does overlay_dx replace RMSE or MAE?
The authors position it as complementary rather than a replacement. On their synthetic data, overlay_dx correlated negatively (-0.62 to -0.5) with MAE/RMSE/MSE and only weakly — slightly positively — with MAPE/NRMSE, meaning it summarizes tolerance-band accuracy and threshold behavior rather than reproducing error-magnitude rankings.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are not analyzable from the extracted text; the Fig. 2 correlation range (-0.62 to -0.5) is taken from §4.2 prose, and overlay-curve shapes in Fig. 4 could not be inspected.
- Metric hyperparameters (max_percentage, min_percentage, step) are absent from the text, so Table 2 scores cannot be re-derived from the paper alone; GitHub code may specify them.
- Minor formula corruption is possible in extraction (Eq. 2's MAPE may lack absolute-value bars as rendered); this was treated as an artifact, not a confirmed error.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.