Building Trust in Artificial Intelligence: A Necessity for Railway Applications
Lefebvre Renard Cl\'ement, L\'eb\'e Vincent, Da Silva Ribeiro Pereira Ricardo, Sundell Johan, Jaoul Arnaud Saiah Kenza, Mijatov\'ic Nenad
cs.AI
Quick verdict
Position paper mapping robustness, ODD and explainability onto railway safety standards; a solid bridge, but no experiments back its 'ready to use' claim.
Summary
This position paper by Alstom researchers argues that deploying AI in safety-critical railway applications requires progress in three fields: robustness, Operational Design Domain (ODD), and explainability. After surveying the regulatory landscape (EU AI Act 2024/1689, ISO/IEC TR 5469, IEC 61508, EN 50126/50716/50129) and the SIL framework with tolerable hazard rates down to 10^-9 per hour, it reviews adversarial robustness (empirical vs. certified defenses, randomized smoothing, Lipschitz networks), explainability (global SHAP and local GradCAM examples drawn from railway studies, with candid caveats that post-hoc explanations mislead under correlated features), and ODD definition and lifecycle per DIN DKE SPEC 99004, linking ODD boundary detection to data-drift monitoring. It then proposes a system-level architecture (Fig. 4) embedding the three pillars in a safe MLOps loop with continuous monitoring and change management. The paper is purely conceptual: no experiments, data, or worked case study substantiate its central claim that non-railway methods are 'ready to use', and the gap between typical 80-99% model accuracy and SIL4 hazard-rate requirements is asserted rather than analyzed. Its main value is as a standards-literate mapping between the ML and railway-safety communities.
Review complete
Assessment at a glance
3
risk flags identified
● high 0
● medium 2
● low 1
4 strengths · 4 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- AI in railways is currently confined to non-safety-critical applications because existing standards and regulations provide no AI-specific compliance route.
- Robustness, Operational Design Domain, and explainability are the three pivotal research fields for trustworthy railway AI, with most algorithms transferable from non-railway domains.
- Integrating the three pillars into a safe MLOps environment with continuous monitoring is a prerequisite for acceptance by regulators, operators, and the public.
- Meeting trust requirements mandated by frameworks such as the EU AI Act will unlock the pace of AI adoption across mission-critical domains.
Strengths
Grounded mapping between ML research and the railway regulatory corpus
Sections 1.1-1.2 connect the EU AI Act risk classes, ISO/IEC TR 5469's three-stage realization principle, and IEC 61508/CENELEC standards to the SIL/THR table (10^-9 to 10^-5 per hour) with concrete examples (Copenhagen Metro ATP functions at SIL4, door management at SIL3).
Unusually candid treatment of XAI limitations
Section 3.2 states that post-modeling explanations are 'a representation of the model's reasoning, not the reality', flags SHAP's unreliability under correlated features, and names interpretable-by-design models as the only currently safe route for decision-making in regulated settings.
Railway-specific rather than generic literature grounding
Identifies the real industrial bottleneck in ODD management
Weaknesses
No empirical content whatsoever
Internal tension between the abstract and the body
The SIL-versus-accuracy gap is asserted, not analyzed
The system approach lacks auditable criteria
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 3 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main argument of this paper?
The authors, researchers at Alstom, argue that AI can reach safety-critical railway applications only if three fields mature together: model robustness (ISO/IEC 24029), explicit definition of the Operational Design Domain (DIN DKE SPEC 99004), and explainability (ISO/IEC 22989) — all integrated into a continuously monitored, safety-aware MLOps process that satisfies regulators under frameworks such as the EU AI Act.
Is this an experimental study?
No. It is a position/review paper: it surveys standards (EU AI Act, IEC 61508, EN 50129, SIL levels) and machine-learning literature (adversarial robustness, certified defenses, SHAP/GradCAM explainability, data-drift detection) and proposes a conceptual system architecture, but reports no experiments, datasets, or code.
Why is AI not used in safety-critical railway systems today, according to the paper?
Because railway safety functions must meet Safety Integrity Levels up to SIL4, tolerating roughly one dangerous failure per 100,000 years (hazard rate on the order of 10^-9 per hour), while typical AI models reach 80-99% accuracy, remain vulnerable to out-of-distribution and adversarial inputs, and lack verification methods recognized by current railway standards.
Which standards and regulations does the paper discuss?
It covers the EU AI Act (Regulation 2024/1689), ISO/IEC TR 5469 on functional safety and AI, IEC 61508, the CENELEC railway standards EN 50126, EN 50716 and EN 50129, ISO/IEC 22989 (AI concepts and terminology), ISO/IEC 24029 (robustness assessment of neural networks), and DIN DKE SPEC 99004 on Operational Design Domains in rail.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are available only as captions in the extracted text; the actual visual content (e.g., the Figure 4 system architecture) could not be assessed.
- This is an arXiv preprint (2609.18278, September 2026) and appears not yet peer-reviewed; standards citations (e.g., DIN DKE SPEC 99004) could not be verified against the source documents.
- Minor internal inconsistency: ISO/IEC 22989 is dated 2022 in Section 3.1 but 2023 in the Section 1.1 standards list.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.