Toward Auditable and Calibrated AI for Dementia-Related Crash Severity Prediction: A Selective Deferral Framework to Support Human Review
Gaurab Chhetri, Anika Baitullah, Subasish Das
cs.AI · cs.HC
Quick verdict
Reframes crash severity as auditable AI triage with leakage control; honest stats, but single split, weak absolute performance, and unevaluated deferred cases.
Summary
This study reframes dementia-related crash severity prediction on Texas CRIS records (4,781 crashes, 2017-2025) as a decision-aware triage problem with three classes (O/BC/KA), emphasizing leakage control, calibration, and selective deferral over raw accuracy. The authors compare structured, TF-IDF narrative, early/late fusion, BERT-family, and local LLM (Gemma, Qwen) models on a stratified 70/15/15 split (718 test cases, 73 KA), additionally masking 22 seed phrases plus up to 75 corpus-derived outcome-revealing narrative expressions. In the reported split, leakage-controlled Gemma attains the highest macro-F1 (0.545; bootstrap CI [0.507, 0.583]) and KA recall 0.630, though paired intervals versus early fusion include zero except for KA recall. Temperature-scaled late fusion reduces ECE from 0.089 to 0.033, enabling confidence-ranked deferral: at 70% coverage, retained-case macro-F1 rises to 0.573 and severity cost falls to 0.577, while deferred cases (including 25 KA) are left unevaluated. Masking only modestly degrades narrative and fusion models, indicating both leakage and genuine textual signal. Main limitations: a single split without ranking uncertainty, retained-only deferral metrics, an author-defined cost matrix, low absolute discrimination (KA PR-AUC 0.308), no human-review evaluation, and record-reported rather than clinically verified dementia status.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Police narratives contain both legitimate crash-sequence signal and outcome-revealing leakage; masking removes much of the latter while performance does not collapse (e.g., late fusion macro-F1 0.539 to 0.522).
- In the reported single split, leakage-controlled Gemma achieves the highest observed macro-F1 (0.545, 95% CI [0.507, 0.583]) and KA recall (0.630), without establishing general model superiority.
- Temperature scaling of late fusion reduces ECE from 0.089 to 0.033 while preserving argmax predictions, making confidence reliable enough to drive selective deferral.
- Selective deferral improves retained-case macro-F1 (to 0.573) and lowers severity cost (to 0.577) at 70% coverage while routing uncertain cases, including many KA records, to a proposed human-review queue.
Strengths
Leakage control as a first-class experimental axis
Paired full-vs-masked evaluation (Table 4, Figure 4) quantifies how much narrative performance rests on outcome-revealing phrases (e.g., 'transported to university hospital'); a rare and appropriate control for crash-report NLP.
Honest statistical self-reporting
The authors state that paired CIs 'include zero' for Gemma vs early fusion (macro-F1 diff 0.022 [-0.026, 0.070]) and explicitly disclaim general superiority; the majority baseline (0.532 accuracy, zero KA recall) anchors the imbalance argument.
Calibration tied to a concrete operational mechanism
Artifact and audit discipline
Robustness check on the cost-sensitive decision rule
Weaknesses
Retained-only deferral metrics
Leakage audit omits the headline models
Single split, weak absolute performance
Author-defined cost matrix
Unquantified cohort construction
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Auditable, Calibrated AI for Dementia-Related Crash Severity Prediction
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is selective deferral in this crash severity paper?
The model ranks its test predictions by calibrated confidence and automatically classifies only the most confident share (100%, 90%, 80%, or 70% coverage); the remainder goes to a proposed human-review queue. Among automatically classified cases, macro-F1 and a severity cost improve as coverage drops — for the calibrated fusion model, cost falls from 0.675 to 0.577 at 70% coverage — but the paper does not evaluate the deferred cases themselves, so no end-to-end benefit after review is demonstrated.
Did police narratives actually help predict crash severity?
Yes, but partly through leakage. Narratives contain outcome-revealing phrases such as 'were transported to university hospital'. After masking 22 seed terms plus up to 75 corpus-derived phrases, narrative logistic regression macro-F1 dropped from 0.516 to 0.502 and late fusion from 0.539 to 0.522. The remaining performance suggests narratives still carry legitimate crash-sequence information beyond the masked outcome words.
Which model performed best for dementia-related crash severity prediction?
In the single reported test split, the locally run LLM Gemma under leakage-controlled masking had the highest macro-F1 (0.545; 95% CI [0.507, 0.583]) and KA recall (0.630). However, paired bootstrap intervals versus early fusion include zero for most metrics, so the authors themselves do not claim general superiority. Calibrated late fusion is recommended for deferral because of its better calibration (ECE 0.033).
Is the dataset used in this study publicly available?
No. The 4,781 dementia-related crashes were derived by the authors from the Texas CRIS database (~5M records, 2017-2025) via keyword search and manual narrative review. CRIS data is request-based, dementia status is record-reported rather than clinically verified, and the paper text states no public code repository, although it describes an extensive retained artifact manifest and a companion web page.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 3 as extracted lacks rows for BERT-Small and for uncalibrated leakage late fusion (the ECE 0.089 value appears only in running text), so some values may be truncated in extraction.
- Figures 1-9 are referenced but only caption text is available; confusion-matrix, reliability-curve, and ECDF details could not be verified from the extraction.
- The companion web page and artifact manifest are described but not inspected; actual artifact or code availability could not be confirmed, hence codeAvailable=false reflects the paper text only.
- Supplementary details (full prompt templates, complete mask-term list, cost-rule derivation) are absent from the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.