Uncertainty-Aware End-to-End AI Weather Forecasting: Disentangling Observation and Model Contributions
Rodrigo Almeida, Noelia Otero, Jost Arndt, Simon Baur, Wojciech Samek, Jackie Ma
physics.ao-ph · cs.LG
Quick verdict
Nested encoder-noise+dropout ensemble makes Aardvark probabilistic with OSE-backed attribution; rigorous and honest, but single-year test and 49 members.
Summary
The paper converts the deterministic Aardvark Weather end-to-end forecasting system into a probabilistic one by attaching a stochastic mechanism to each component: learned, input-dependent Gaussian noise at the observation encoder and Monte Carlo dropout (p=0.05) in the processor, finetuned with a fair-CRPS objective on top of the pretrained weights (~65 A100-hours). Sampling M=7 encoder draws times N=7 dropout rollouts yields a 49-member nested ensemble whose variance splits exactly into encoder and dropout components via a law-of-total-variance (one-way ANOVA) decomposition. Probabilistic finetuning improves the ensemble-mean RMSE by 4.2% on average over 60 variable-lead combinations (up to 16%), improves fair CRPS by 24-38%, and yields near-calibration against ERA5 for 2018 (spread-skill ratio 0.98 over days 1-10), while trailing the operational IFS ENS (CRPS gap ~80% at day 1, narrowing to 18-36% by day 10). An observation-denial experiment supports the attribution: withholding the IASI sounder roughly doubles the encoder variance at days 1-3 without moving the dropout component, while denying radiosondes changes nothing. Main limitations: the encoder branch is strongly under-dispersive in isolation (SSR 0.46 to 0.19), the noise-mask interaction folds into the dropout term, the encoder-variance estimator's non-negativity floor is occasionally active, verification covers a single year and a single backbone, and the station comparison uses a weaker modular baseline.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Probabilistic finetuning of the pretrained Aardvark system improves the mean forecast, reducing ensemble-mean RMSE by 4.2% on average across 60 variable-lead combinations (95% CI [-4.9, -3.4]%, significant for 50/60) and fair CRPS by 24-38%.
- The 49-member nested ensemble is calibrated against ERA5 through the medium range (variable-mean spread-skill ratio 0.98 over days 1-10) with no spread inflation or statistical post-processing.
- A law-of-total-variance decomposition assigns most predictive variance to the dropout branch (share 0.86 at day 1 rising to 0.96 at day 10), while the encoder branch stays nearly flat with lead time.
- The attribution passes a causal cross-check: withholding the IASI sounder roughly doubles the encoder variance at days 1-3 (+96 to +111%) while the dropout component responds by at most 8%, and denying IGRA radiosondes leaves every component within 1% of baseline.
- The uplift is cheap (~65 A100-hours, no new model or generative treatment), and denial experiments cost a few GPU-hours, enabling observation-network design queries at inference cost.
Strengths
Falsifiable attribution test, not just a declared split
The observation-denial cross-check (Fig. 5b,c) turns the aleatoric reading into a testable prediction: denying the most influential stream (IASI) inflates the encoder variance by +96 to +111% at days 1-3 while Udrop never moves by more than 8% (largest -8% at day 2), and denying the least influential stream (IGRA radiosondes) leaves every component within 1% of baseline, over 24 initializations with common random numbers.
Verification statistics above field standard
Fair (finite-ensemble-corrected) CRPS throughout, WeatherBench 2 latitude weighting, per-initialization aggregation (n=337) before testing, circular moving-block bootstrap (5x10^3 resamples) with a data-adaptive block length, and Diebold-Mariano tests with Bartlett HAC variance and the Harvey-Leybourne-Newbold correction; nonlinear aggregates are recomputed on each resample rather than approximated.
Interpretive discipline unusual in UQ papers
Cheap uplift that also improves the mean
Reproducibility artifacts
Weaknesses
Encoder branch is strongly under-dispersive in isolation
Small nested design and a clipping estimator at long leads
Station baseline is weaker than the published configuration
Interaction term folded into the dropout axis
Narrow verification window and a selection/evaluation mismatch
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a genuine gap: end-to-end (E2E) AI weather systems such as Aardvark Weather and ECMWF's GraphDOP forecast directly from raw observations and skip data assimilation, but prior work left them deterministic. The authors uplift the public Aardvark model into a probabilistic 'E2E ensemble' with exactly one stochastic mechanism per component - learned, input-dependent Gaussian noise at the observation encoder (intended aleatoric), Monte Carlo dropout at p=0.05 in the processor (intended epistemic), and a deterministic frozen decoder - and pose three questions: where the spread originates, whether the two contributions can be disentangled, and what the uplift costs. Central claims: (i) probabilistic finetuning improves the mean forecast by 4.2% RMSE on average across 60 variable-lead combinations (95% CI [-4.9, -3.4]%, significant for 50 of 60, up to 16%); (ii) the ensemble is calibrated through the medium range (variable-mean SSR 0.98 over days 1-10) with no post-hoc inflation; (iii) a law-of-total-variance decomposition, cross-checked by observation denial, attributes spread to the two sources. Claims are, with few exceptions, matched to the evidence, and the interpretive hedging (mechanism names first, aleatoric/epistemic reading only after the test) is exemplary.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Aardvark Weather and what does this paper add to it?
Aardvark Weather is an AI system that produces global gridded and station forecasts directly from raw Earth observations (satellites, stations, ships, balloons), replacing the numerical weather prediction pipeline including data assimilation - but it issues only a single deterministic forecast. This paper adds calibrated uncertainty by attaching learned input-dependent noise to its observation encoder and Monte Carlo dropout to its forecast processor, and shows how to split the resulting ensemble spread into an observation-driven and a model-driven part.
How close is this end-to-end ensemble to ECMWF's operational forecasts?
It still trails the operational IFS ensemble in probabilistic skill at every lead time: the CRPS gap is about 80% in the variable mean at day 1, narrowing to 18-36% by day 10. The authors attribute part of this gap to the deliberately lean backbone (a 1.5-degree grid and about 0.73 TB of observations from nine streams, with no aircraft or radio-occultation data).
What do the encoder branch and dropout branch mean in the uncertainty decomposition?
The encoder branch is the spread created by learned, input-dependent Gaussian noise at the observation-assimilation step, intended as observation-driven (aleatoric) uncertainty; the dropout branch is the spread from Monte Carlo dropout in the forecast processor, intended as model-driven (epistemic) uncertainty. A nested 7x7 ensemble makes them separately measurable via a variance decomposition, and observation-denial experiments (removing the IASI sounder doubles the encoder variance while dropout barely moves) support this reading. The authors stress the reading is component-aligned attribution, not recovered ground truth.
Are the code, weights and data for this study available?
Yes, as stated in the paper: the finetuned model weights and evaluation data are on Hugging Face (rodrigoalmeida1994/uqe2e), the code is on the authors' Fraunhofer HHI GitLab, and the training/verification inputs are the publicly released Aardvark dataset plus ERA5 and IFS ENS fields fetched via WeatherBench.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels are present only as captions and axis descriptions in the extraction; graphical content of Figs. 1-5 and S1-S2 could not be visually verified, so all numeric readings rely on captions and body text.
- The extracted text ends with Supplementary Table S1 (page 17/17), so the main text and listed supplementary notes appear complete; no truncation is apparent.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.