SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment
Chenxi Wu, Zimu Wang, Haiyang Zhang, Wei Wang, Zhijie Xu
cs.CL
Quick verdict
First industrial ISO 26262 HARA benchmark (3k real cases, expert-validated LLM judge); LLMs fail ASIL classification (F1 0.261) but stats are thin.
Summary
The paper introduces SAFARI, described as the first industrial benchmark for LLM-assisted automotive Hazard Analysis and Risk Assessment (HARA) under ISO 26262. It contains 3,000 de-identified HARA cases drawn from real industrial projects, all anchored to a single failure mode (unintended drive force) while varying operational context. Task 1 requires generating hazardous event, hazard, and persons at risk; Task 2 requires predicting Exposure, Severity, Controllability, derived ASIL, Safety Goal, and FTTI. Open-ended outputs are scored by a reference-anchored LLM-as-a-judge protocol along four dimensions, validated against certified engineers on 180 samples (Spearman's rho 0.807-0.951), with a cross-family check showing a maximum correlation difference of 0.027. Experiments on nine frontier LLMs under few-shot and chain-of-thought prompting find plausible hazard narratives but weak risk classification: best ASIL macro-F1 is 0.261. CoT generally degrades categorical assessment - the false-QM rate rises from 14.9% to 41.0% - while improving FTTI estimation. A transparent rule-based baseline beats 13 of 18 LLM configurations on Controllability. Error analysis localizes failures to scenario-context omission and controllability misjudgment. Limitations include a single malfunction class, severely skewed labels (QM 81.7%, ASIL D absent), absence of significance testing, and references that may require context absent from model inputs.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SAFARI is the first industrial benchmark for LLM-assisted HARA under ISO 26262, built from 3,000 de-identified real-workshop cases.
- A reference-anchored LLM-as-a-judge protocol reaches high expert correlation (Spearman ρ 0.807–0.951) and is robust to judge model family (maximum difference 0.027).
- Frontier LLMs generate plausible hazard narratives but fail standards-grounded classification, with the best ASIL macro-F1 of only 0.261.
- CoT prompting generally degrades categorical E/S/C/ASIL assessment (false-QM rate rising from 14.9% to 41.0%) while improving FTTI estimation.
- Failures localize to scenario-context omission during hazard generation and controllability misjudgment during risk assessment.
Strengths
Real industrial provenance with a documented QA chain
Cases come from structured HARA workshops by certified engineers (§3.1), with calibration, dual independent review (~90% raw pre-adjudication agreement), senior adjudication, and verification that every (E,S,C) tuple maps to the correct ASIL per ISO 26262-3 (§3.3).
Judge protocol validated with a cross-family bias control
Qwen3.6-Max judge reaches Spearman ρ 0.807–0.951 against experts on 180 outputs (Table 2); a GPT-5.5 second-judge check bounds same-family bias at a maximum correlation difference of 0.027 (§4.1).
Honest negative results against a transparent non-LLM comparator
Concrete, cascade-aware error localization
Workflow-faithful task decomposition
Weaknesses
Extreme label imbalance interacts unexplained with macro-F1
No inferential statistics or variance control
Thin judge validation and an unexplained HLM floor
References may require context absent from model inputs
Narrow and underused design
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SAFARI (arXiv 2609.20584) is a benchmark paper at the intersection of NLP and automotive functional safety. It makes three claims: (1) the first industrial benchmark for LLM-assisted HARA under ISO 26262, built from 3,000 de-identified cases produced in real HARA workshops; (2) a reference-anchored LLM-as-a-judge protocol validated against certified engineers; (3) evidence that frontier LLMs write plausible hazard narratives but fail standards-grounded risk classification (best ASIL macro-F1 0.261), with CoT mostly harmful except for FTTI. Scope is deliberately narrow: one failure mode (unintended drive force/torque), 11 structured inputs, 9 outputs (Table 1), splits 1,600/400/1,000.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SAFARI benchmark?
SAFARI is a collection of 3,000 de-identified, real-industry automotive hazard-analysis cases (ISO 26262 HARA) used to test how well large language models can write hazard descriptions and assign risk ratings (Exposure/Severity/Controllability and the derived ASIL). It is split into a generation task (hazardous event, hazard, persons at risk) and a classification task (E/S/C, ASIL, safety goal, FTTI).
How well do LLMs perform on ISO 26262 risk assessment in this study?
Poorly on classification: across nine frontier models the best ASIL macro-F1 was only 0.261, and a simple transparent rule-based baseline beat most models on Severity and Controllability. Models were noticeably better at writing plausible hazard narratives than at assigning correct risk classes.
Does chain-of-thought (CoT) prompting help with hazard analysis?
Mostly no. CoT generally degraded categorical E/S/C/ASIL ratings — the rate of wrongly assigning QM to reference ASIL A–C cases rose from 14.9% to 41.0% — and lowered judge scores on most text dimensions. The one consistent benefit was better FTTI (fault tolerant time interval) estimation.
Is the SAFARI dataset publicly available?
Yes, the paper links a GitHub repository (github.com/xixi47520-hash/HARA) containing the 3,000 de-identified cases. The paper text does not state a license, nor whether evaluation prompts, judge rubrics, or scoring code are also released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Tables 3–4 column headers are garbled in extraction; Table 4 column semantics were inferred by cross-checking Appendix Table 5 (E/S/C/ASIL macro-F1, then ASIL accuracy, SGA, FTTI-MAE).
- Figures 2–5 are recoverable only as inline numeric text; bar details and bolding were not independently verifiable.
- Judge prompts, rubrics, and API prompt templates are absent from the extracted text; protocol reproducibility is judged from description only.
- Review based on arXiv v1 main text plus Appendices A–B; any supplementary files were not seen.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.