Protocol before progress: leakage-aware evaluation of AIS trajectory prediction
Zobeir Raisi, Vali Mohammad Nazarzehi Had
cs.LG
Quick verdict
Claims protocol, not architecture, drives AIS prediction gains: oracle 2.1–3.2×, vessel leakage 23–25%, region collapse; rigorous two-corpus audit, convincing.
Summary
This paper audits whether reported gains in AIS vessel-trajectory prediction come from better models or from permissive evaluation protocols. The authors build three disjoint split manifests — by vessel, by time, and by region — over two corpora: 31 days of Danish Maritime Authority AIS and 30 days of NOAA Houston–Galveston AIS. They reimplement TrAISformer and GATransformer, reconstruct the paywalled AISFormer and MGFormer under controlled assumptions, and add a compact 0.43M-parameter in-house encoder. Three effects replicate on both corpora. First, best-of-16 oracle decoding lowers TrAISformer's error 2.1–3.2× relative to greedy decoding, and a noise-perturbed constant-velocity control given the same oracle captures roughly three fifths of that gain in log terms. Second, sharing vessels across the split lowers the large models' greedy error by 23–25%, against 2% or less for the small encoder; a causal test attributes the gain to the test vessels' own training history, and the effect grows with model capacity. Third, region-disjoint splits collapse the absolute-position tokenized models (2.2 to 24.6 km at one hour on the US corpus) because 99.9% of test longitude bins are absent from training. GATransformer's graph attention shows no measurable benefit, while its waterway feature is worth 12–22%. The time-disjoint effect does not replicate across corpora (13% versus 2%). Limitations include reconstruction-based comparators, a single-seed largest capacity point, and one longitude cut per corpus.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Best-of-16 oracle decoding lowers TrAISformer's error by a factor of 2.1–3.2 versus greedy decoding, and roughly three fifths of that benefit (in log terms) would accrue to a noise-perturbed constant-velocity control that learns nothing.
- Splits that share vessels lower the greedy error of 7–8M-parameter token models by 23–25%, versus 2% or less for a 0.43M encoder; a causal test attributes the gain to the test vessels' own training history, and the effect grows with model capacity.
- Region-disjoint evaluation collapses absolute-position tokenized models (2.20 to 24.59 km at one hour on the US corpus) because 99.9% of test-context longitude bins never occur in training, while a local-offset encoder degrades by a bounded amount.
- GATransformer's graph attention provides no measurable benefit on either corpus, while its engineered waterway feature changes error by 21.9% (Danish) and 11.8% (US).
- The time-disjoint split effect is corpus-dependent (13% Danish versus 1.8% US greedy error reduction) and should be reported as such rather than assumed to replicate.
Strengths
Oracle-decoding control separates metric artifact from learned skill
The constant-velocity-plus-noise control (Sec. 4.3, 5.1) gains a factor of 1.6 from best-of-16 selection (3.56 to 2.21 km at one hour) without reading the trajectory, cleanly partitioning TrAISformer's 2.1× oracle factor into protocol reward for diversity versus genuine learned skill — a measurement the sampled-forecast literature it cites (Sec. 2.3) lacks.
Causal attribution rather than observational split comparison
Table 11 holds the test set fixed and swaps equal numbers of training windows for the test vessels' own non-overlapping history: greedy error falls −0.292±0.045 km at 100% of H, a clean-reduced control shows no data-quantity effect (+0.079±0.023 km), and a quarter of the history already yields −0.230 km. This is attribution, not correlation.
Reproduction fidelity is demonstrated, not asserted
Mechanism verified, not just correlation
Disciplined uncertainty reporting
Weaknesses
Two of four comparators are reconstructions, not reimplementations
Time-disjoint headline uses raw km despite the paper's own normalization argument
Capacity–leakage trend's endpoint is single-seed
Second-corpus replication is incomplete
Single longitude cut per corpus; one month per corpus
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is data leakage in AIS vessel trajectory prediction?
It occurs when information from test vessels reaches training — for example, a random or chronological split places windows from the same ship on both sides, letting the model exploit that ship's habits. This paper measures the effect on two corpora: sharing vessels across the split lowers the greedy-decoding error of 7–8M-parameter trajectory transformers by 23–25%, while a compact 0.43M-parameter encoder gains 2% or less.
Why is best-of-N decoding considered misleading for trajectory prediction?
Best-of-N keeps whichever of N sampled predictions lands closest to the ground truth, so the selection consults the answer. The paper shows a constant-velocity predictor perturbed by noise — which learns nothing from the trajectory — still improves by a factor of 1.6 (3.56 to 2.21 km at one hour) under the same oracle. It recommends always reporting single-shot (greedy) error alongside any best-of-N figure.
Which train/test split should vessel trajectory prediction use?
The paper's protocol uses three separate manifests per dataset: vessel-disjoint (no MMSI in more than one split) as the primary evaluation, a chronological split to test temporal generalization, and a region-disjoint split as a transfer stress test. It also recommends stating where any fitted auxiliary structure (waypoint graphs, codebooks, normalizations) drew its data from, since split manifests do not govern that.
Does graph attention improve vessel trajectory prediction?
In this audit on Danish and US Gulf AIS resampled to 10 minutes, removing GATransformer's spatial graph-attention encoder changed error by −2.1% to −2.7% — within seed noise — across a 4.7-fold range of neighborhood densities, while the engineered distance-to-waterway-intersection feature was worth 12–22%. The authors caution the result is specific to their reconstruction, region, and sampling cadence.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–8 appear in the extraction only as axis-label fragments; per-figure trends were assessed from surrounding prose and tables, not plotted data.
- Published TrAISformer table values and the paywalled AISFormer/MGFormer contents are cited secondhand and could not be verified from the provided text.
- The 67% US neighbor-provenance statistic appears only in the contribution list; it may derive from material not included in the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.