Accountable and uncertainty-aware evaluation of sensor-based AI under distribution shift: devices, subjects, and nearly three years underground
Benny Platte, Rico Thomanek, Christian Roschke, Marc Ritter
cs.LG · stat.ME · stat.ML
Quick verdict
Proposes a staged, quantile-based evaluation protocol for sensor AI under distribution shift, convincingly shown over 34 months in two mines; data not public.
Summary
The paper proposes a staged evaluation protocol that treats the validation of a deployed sensor-AI model as a measurement with declared reference levels and quantified uncertainty. Four cumulative stages hold out devices, subjects, and recording epoch, with the customary random train-test split demoted to a declared upper bound; roll-out is decided on 5% quantiles of repeated trainings against class-count-correct chance references, guarded by an out-of-present-scope (OOPS) rate. The protocol is demonstrated on LSTM-based geomagnetic localisation in two German underground mines (7,003 training runs, 3,237 GPU-hours). On matched configurations the random split overstates macro precision by +0.14 (cross-device) and +0.41 (cross-subject), driven by up to 99.86% window-overlap leakage. Re-evaluating unchanged models 34 months later on an unseen device generation and surveyor, the 5% quantile of present-conditioned precision is 0.39 over 299 runs (16.5 times chance); of the apparent 47-point degradation for the best representation, 23.5 points are exact class loss and 21.4 are annotation-boundary ambiguity (16.5-25.8 across sets), bounding genuine signature change. A bimodal configuration passes a mean-based t-test while its 5% quantile lies below chance, motivating quantile verdicts. Limitations: two persons per mine, device-person confounding with unlogged held-out units in the uranium mine, only two validation walks at the cross-time stage, and code/data available only on request.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Matched-configuration comparisons show the random train-test split overstates macro precision by +0.14 under device change and +0.41 under subject change, driven by up to 99.86% window-overlap leakage (sections 5.1-5.2).
- Unchanged models re-evaluated 34 months later on an unseen device and surveyor retain a 5%-quantile present-conditioned precision of 0.39 over 299 runs, 16.5 times the 1/42 chance level (section 7.1, Table 5b).
- Of the apparent 47-point long-term degradation for the best representation FV, 23.5 points are exact class-inventory loss and 21.4 points are annotation-boundary ambiguity, bounding genuine signature change between 2.1 and 23.4 points (section 6).
- Mean-based acceptance misclassifies deployment risk: a bimodal FK configuration passes a mean t-test (p of order 1e-13) while its 5% quantile equals 0.03 times chance (section 7.1).
- A guarded quantile-based roll-out verdict is affordable at 22.0 GPU-h (0.68% of the full study) and supplies measurable counterparts to EU AI Act Articles 9, 15 and 72 (sections 7.3, 8.2).
Strengths
Genuinely novel longitudinal evidence
The 34-month cross-time stage (section 6, Table 4) re-evaluates unchanged models on an iPhone 16 Pro that postdates training and a held-out surveyor; the authors' literature review (2.1-2.2) plausibly supports the claim that no comparable geomagnetic re-evaluation exists. The 5% quantile of present-conditioned precision is 0.392 over 299 runs, 16.5x the 1/42 chance level.
Exceptional disclosure discipline
Undocumented 0.8/0.7 multiplicative artefacts on FA+L metric columns were found, reversed, and verified against stored confusion matrices to below 1e-12 (4.4); a systematic -1.1 pp pipeline re-scoring offset (79 archived models, r = 0.999) is quantified and carried into all long-term comparisons (6); missing optimiser, normalisation and seeds in the archived logs are stated outright (4.4).
Statistically mature reporting
Degradation decomposition is verifiable, not narrative
Actionable decision economics and regulatory anchoring
Weaknesses
Uranium-mine replication is confounded and partially unlogged
Thin cross-time evidence base and mixed metric conventions
Cross-subject verdicts rest on fragile bases
Access-gated reproducibility
Underpowered ranking conclusions relative to abstract-level phrasing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses a gap it documents convincingly: the fingerprint-localisation literature evaluates within a single measurement campaign, so performance under device replacement, personnel change and multi-year operation is simply unreported (sections 2.1-2.2). It contributes a staged evaluation protocol framed explicitly as measurement (GUM/JCGM 100, VIM, JCGM 106) and makes five claims: (1) a staged protocol with quantile-based roll-out decisions and an OOPS guard (section 4, Table 2); (2) quantified random-split optimism of +0.143/+0.413 on matched configurations, with the window-overlap leakage mechanism (5.1-5.2); (3) a 34-month cross-time evaluation with a three-part degradation decomposition (section 6); (4) a three-step one-shot risk assessment showing mean-based reporting misclassifies a bimodal configuration (section 7); (5) representation findings that attitude channels help in-distribution but hurt under shift, and that random-split rankings do not predict shift rankings (5.4). Claims are scoped to one case study - LSTM geomagnetic localisation in two German mines, 7,003 runs - with an explicit transferability argument (8.2) and an EU AI Act anchoring the authors carefully disclaim as non-legal.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why is a random train-test split misleading for time-series sensor models?
Because sliding windows cut from the same recording overlap heavily: at a 700-sample window with step size 1 the overlap is 99.86%, so near-duplicates of most test windows sit in the training set. This paper measures the resulting inflation at +0.14 macro precision when the device changes and +0.41 when the person changes, and recommends reporting window overlap and using the random split only as a declared upper bound.
How well does smartphone geomagnetic indoor positioning hold up over years?
In this study, LSTM models trained underground in a German mine still achieved a 5%-quantile precision of 0.39 on the 42 location classes still reachable 34 months later - 16.5 times the random-guess level - using an unseen phone model and an unseen surveyor. Much of the apparent loss is bookkeeping: 23.5 points came from classes no longer accessible and roughly 16-26 points from shifted annotation boundaries rather than from real change in the magnetic signatures.
What is the OOPS rate in this paper?
OOPS stands for out-of-present-scope rate: the share of a model's predictions that land on location classes which no longer exist at deployment time. Present-class-restricted scores ignore these predictions entirely, so OOPS exposes silent misdirection - for example, one feature family looked competitive on present classes while directing 41-43% of predictions to vanished classes, versus 9.5-18.4% for the alternatives.
What does this evaluation protocol have to do with the EU AI Act?
The authors map their staged verdicts, declared reference levels and cross-time re-evaluation onto Articles 9, 15 and 72 of Regulation (EU) 2024/1689, which require risk-management testing against probabilistic thresholds, declared accuracy and robustness levels, and post-market monitoring for high-risk AI systems. They present this as an anchoring only and explicitly exclude any legal assessment.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are present only as captions in the extracted text; every number cited in this review was taken from body text and tables (e.g., Tables 2-8), not from reading plot content.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.