A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out
Mahdi Naser Moghadasi, Faezeh Ghaderi
cs.LG
Quick verdict
Claims pretraining familiarity survives a contamination-free hold-out; credible protocol and honest stats, but corpus cause rests on one model-domain match.
Summary
The paper builds a forecasting benchmark whose test window postdates every evaluated model's release, so test observations cannot have been memorised. Thirteen forecasters — four classical (SeasonalNaive, Theta, AutoETS, AutoARIMA), three trained per dataset (LightGBM, LSTM, N-BEATS) and six pretrained (Chronos-Bolt small/base, Chronos-2, TimesFM, TimesFM-3, Moirai-2) — are evaluated on seven groups from five keyless, continuously published sources (Wikipedia pageviews daily/weekly/monthly, hourly ERA5 weather and CAMS air quality, Danish electricity, ECB exchange rates), with the origin fixed at 1 January 2026. By average rank with Nemenyi intervals, pretrained models win five of seven groups; Theta wins electricity and nothing beats seasonal naive on exchange rates. The planned intrinsic explanation fails: neither seasonal strength nor spectral entropy orders the pretrained advantage. The authors instead attribute the gradient to corpus familiarity: the largest gain (28% lower MASE on weekly Wikipedia) falls exactly where TimesFM's corpus dominates, and the TimesFM family outranks the Chronos family by -0.53 ranks on Wikipedia versus -0.09 elsewhere (Mann–Whitney p<10⁻⁵, rank-biserial -0.13). All pretrained models under-cover their 80% intervals while classical methods over-cover; AutoARIMA costs a median 54× more per series. Limitations: a single forecast origin, two small panels, and a single-corpus confound the authors acknowledge.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A hold-out whose test window postdates every evaluated model's release eliminates test-set memorisation, yet under it pretrained models win only 5 of 7 groups, lose electricity to Theta, and are statistically indistinguishable from seasonal naive on daily exchange rates.
- Neither seasonal strength nor spectral entropy measured on the input window (Table 3) explains where the pretrained advantage appears; seasonal strength is if anything negatively associated with it.
- The advantage gradient tracks corpus familiarity: the largest gain (28% lower MASE than the best classical method on weekly Wikipedia) coincides with TimesFM's dominant pretraining domain, and the TimesFM family outranks the Chronos family by −0.53 ranks on Wikipedia versus −0.09 elsewhere (Mann–Whitney p<10⁻⁵, rank-biserial −0.13).
- All six pretrained models systematically under-cover their nominal 80% intervals in every domain while the automatic classical methods over-cover (§4.6, Table 4), a family-split calibration failure unreported by leaderboards.
- Benchmarks must therefore hold out domains stated relative to disclosed pretraining corpora, not merely dates; temporal recency alone cannot separate generalisation from domain familiarity.
Strengths
Genuinely contamination-free evaluation design
Forecast origin fixed at 1 Jan 2026, after the release of all six pretrained checkpoints (§3.1); five keyless, continuously published institutional sources (§3.2); fetchers with manifests and SHA-256 checksums plus a generated Croissant record released rather than frozen data (§8).
Statistical honesty rare in this literature
Refuses to count a rank-first finish under a non-rejecting Friedman (6-of-7 vs the defensible 5-of-7, §4.1); primary comparison via Nemenyi intervals (Table 2) with Holm-corrected Wilcoxon (§3.4); the Danish hydro exclusion shows one of 47 series moved mean MASE six orders of magnitude while ranks stayed unchanged (§5).
Negative result measured, not asserted
Within-family comparison cancels series difficulty
Failure modes and checks disclosed
Weaknesses
Single-corpus confound under the headline mechanism
'Intrinsic explanation fails' generalizes from two candidates
Thin panels and a single forecast origin
Per-dataset baselines under-specified
Aggregation ambiguities in cost and coverage claims
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a concrete evaluation-theory problem: every standard time-series benchmark (M4, ETT, Monash) predates the foundation models scored on it, so published zero-shot numbers cannot separate generalisation from memorisation. The authors build a hold-out whose entire test window postdates all six pretrained checkpoints (origin fixed at 1 January 2026, §3.1) across seven groups from five keyless institutional sources (§3.2). Central claims: (1) under this protocol pretrained models win 5 of 7 groups, lose electricity to Theta, and tie seasonal naive on exchange rates (Table 2); (2) intrinsic series properties — seasonal strength and spectral entropy (Table 3) — fail to explain the pattern; (3) corpus familiarity does: the largest gain (28% lower MASE than the best classical method on weekly Wikipedia; 1.073 vs Theta's 1.499 in Table 1) lands on TimesFM's dominant corpus domain, and the TimesFM family beats the Chronos family by −0.53 average ranks on Wikipedia versus −0.09 elsewhere (1,500 vs 754 series, Mann–Whitney p<10⁻⁵, rank-biserial −0.13, §4.4); (4) benchmarks therefore need domain hold-outs stated relative to disclosed corpora.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'a later test set is not a new domain' mean?
Moving the test window to after a model's release prevents the model from having memorised the exact test observations, but it does not stop the model from having learned the kind of data (the domain) during pretraining. For example, TimesFM was pretrained largely on Wikipedia pageview series, so brand-new 2026 pageviews still resemble its training data. The paper shows this 'domain familiarity', not test leakage, explains much of the pretrained models' advantage.
Did pretrained time-series models beat classical baselines on uncontaminated data?
Partly. On seven test groups drawn entirely after January 2026, pretrained models (Chronos-Bolt, Chronos-2, TimesFM 2.5/3.0, Moirai-2) ranked first in five groups with statistically resolvable wins. However, a classical Theta baseline won the hourly electricity group, and on daily exchange rates every method — including a seasonal naive forecast — was statistically indistinguishable.
Are time-series foundation models overconfident about their uncertainty?
In this study, yes, systematically: all six pretrained models' nominal 80% prediction intervals covered fewer than 80% of actual outcomes in every domain tested, while classical automatic methods covered more than nominal. The authors argue this calibration gap matters more than the small accuracy differences for anyone sizing buffers, reserves, or inventory.
Is the benchmark and code available?
The authors release data fetchers (not the data itself), the evaluation harness, per-series losses and analysis scripts at github.com/mahdinaser/tsfm-bench. Every dataset rebuilds from institutional sources without API keys, and the forecast origin is pinned in the harness so later rebuilds remain reproducible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 appear only as captions in the extracted text; the plotted content (interval spans, cost-accuracy cluster positions) could not be inspected.
- The referenced GitHub repository (tsfm-bench) and released artifacts could not be verified from the text alone; availability claims rest on §8 and the Reproducibility section.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.