Evaluating Accuracy and Probabilistic Reliability of Zero-Shot Time Series Foundation Models
Panagiotis Michael, Moysis Symeonides, Demetris Trihinas
cs.LG · cs.AI
Quick verdict
Zero-shot benchmark of six TSFMs on three datasets finds an accuracy-calibration trade-off; useful but single series per domain, no significance tests.
Summary
This benchmark study evaluates six time series foundation models (TSFMs)—Chronos-2, TiRex, Moirai-2.0, Sundial, TimesFM, and Toto—for zero-shot forecasting against ARIMA, running-average, and random-walk-with-drift baselines plus a supervised PatchTST, on three univariate series: EdgeTraffic vehicle counts, Poland grid load (MixGridPL), and SPY daily prices. Rolling-window experiments vary context length and horizon, measuring point accuracy (sMAPE) and 80% interval reliability (ICE, IMAE). TSFMs generally match or beat statistical baselines, with the largest margins on seasonal MixGridPL (best TSFMs 3.92–3.95% sMAPE at H=12 versus 14% for ARIMA; Chronos-2 6.4% versus 19% at H=672). On EdgeTraffic long horizons, TimesFM and Moirai-2.0 lead on sMAPE but their calibration collapses (ICE 0.73 and 0.56 at H=1800), whereas TiRex and Chronos-2 keep ICE below 0.06; xLSTM-based TiRex is the most consistently calibrated across datasets. The central contribution is documenting an accuracy-versus-calibration trade-off and context-saturation effects. Main limitations: each domain is represented by a single time series; no significance testing or confidence intervals; the supervised PatchTST baseline is deliberately minimized (one layer, hidden dimension 32) for latency parity, weakening claims of supervised-model inferiority; SPY data (1993-2025) likely overlaps TSFM training corpora; results appear only as figures, with tables deferred to the public repository.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Six TSFMs match or exceed statistical baselines and a supervised PatchTST across traffic, energy, and financial series, with the largest gains on seasonal data (§4.1).
- Point accuracy and probabilistic reliability trade off: sMAPE leaders TimesFM and Moirai-2.0 become severely over-confident at long horizons (ICE 0.73 and 0.56 at H=1800 on EdgeTraffic), while Chronos-2 and TiRex keep ICE below 0.06 (§4.2).
- xLSTM-based TiRex provides the most robust interval calibration across horizons and modalities (ICE 0.01–0.04 on MixGridPL, 0.01 with IMAE 0.34 on SPY at C=509).
- Transformer-based TSFMs exhibit context-saturation points beyond which additional history no longer improves, or slightly degrades, zero-shot accuracy (e.g., TimesFM 21%→22.5% past C=1260 on EdgeTraffic).
Strengths
Reliability axis beyond point accuracy
ICE/IMAE (§3.5, eq. 2–3) expose failures invisible to sMAPE: TimesFM/Moirai-2.0 lead EdgeTraffic sMAPE yet reach ICE 0.73/0.56 at H=1800 with IMAE 2.8/3.2, while Toto's Student-T mixture head holds ICE 0.07 (§4.2).
Leakage-aware dataset curation
EdgeTraffic (TDIS '26) and MixGridPL (2024–25) postdate TSFM training corpora (§3.1), supporting genuine zero-shot claims; the benchmark is open-source (github.com/unic-ailab/TSFM-Eval) with reproduction instructions.
Architecture-controlled model span
Controlled scenario design
Honest mixed results
Weaknesses
Single series per domain
No statistical inference
Handicapped supervised baseline
SPY contamination risk
Figure-only reporting and calibration/sharpness conflation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper benchmarks zero-shot forecasting by six TSFMs (Chronos-2 120M, TiRex 35M, Moirai-2.0 11.4M, Sundial 128M, Toto 151M, TimesFM 200M) against ARIMA, running average (RA), random walk with drift (RWD), and a supervised PatchTST, on three univariate series: EdgeTraffic (7,478 obs @1s), MixGridPL (12,407 obs @1h, 2024–25), and SPY (8,324 daily closes, 1993–2025). Central claims: (1) TSFMs match or exceed statistical and supervised baselines; (2) point accuracy and probabilistic reliability trade off — sMAPE leaders TimesFM/Moirai-2.0 become over-confident at long horizons (ICE 0.73/0.56 at H=1800 on EdgeTraffic); (3) xLSTM-based TiRex calibrates most robustly; (4) transformers show context-saturation points. The evidence covers only these three series, though the abstract's framing ('TSFMs outperform statistical methods and supervised models') reads broader than what three single series can support.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper on zero-shot time series foundation models?
Across three datasets (road-traffic counts, Polish grid load, and SPY stock prices), the study finds that TSFMs generally beat statistical baselines on point accuracy, but the sMAPE leaders (TimesFM, Moirai-2.0) become over-confident at long horizons — their 80% intervals miss up to 73% of points — while TiRex (xLSTM-based) and Chronos-2 stay well calibrated. The paper's message is that point accuracy and interval reliability must both be considered when choosing a model.
Which time series foundation model is best for zero-shot forecasting according to this benchmark?
No single model dominates everything. Chronos-2 is judged the most balanced (competitive accuracy plus well-calibrated intervals), and TiRex the most consistently calibrated across all three datasets; TimesFM and Moirai-2.0 often lead on sMAPE but show severe calibration collapse at long horizons.
What datasets and metrics does the TSFM-Eval benchmark use?
Three univariate series: EdgeTraffic vehicle counts (1-second resolution, Iasi, Romania), MixGridPL Polish national grid load (hourly, 2024–25), and Yahoo! SPY ETF daily closes (1993–2025). Point accuracy is measured with sMAPE, and 80% prediction-interval quality with Interval Coverage Error (ICE) and Interval Mean Absolute Error (IMAE), across varied context lengths and forecast horizons.
Is the code and data for this TSFM benchmark publicly available?
Yes. The authors state the benchmark is open-source at github.com/unic-ailab/TSFM-Eval with full tabular results and reproduction instructions. The six models come from Hugging Face in default configurations, and the datasets are described in publicly available sources.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All quantitative results are presented as figures (Figs. 3–5), which could not be analyzed from the extracted text; per-point values rely on numbers quoted in the prose.
- No result tables appear in the manuscript; exact per-configuration values are deferred to the external repository.
- The number of evaluation windows and any repetitions per configuration are not stated, so the variance of reported metrics cannot be assessed.
- Extraction otherwise appears complete (abstract through references); no obviously truncated sections were detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.