Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)
cs.CL
Quick verdict
Audits a typed model coding 195,857 Texas crash narratives for $55.79 with review budgets; strong methods, but human labels are thin on rare variables.
Summary
The paper formulates coding of police crash narratives as gated, typed decisions answered by Jev, a commercial System One model that returns probabilities over fixed options without generating any text. Using Texas CRIS data (2017–2025), the authors screen 499,500 narratives and fully code 195,857 with a 27-question schema at a measured cost of $0.154 per thousand narratives (about $774 for the 5.02M corpus), showing spend is governed by schema size rather than narrative length. Probabilities are audited against coded CRIS fields and against 2,416 blinded human judgments carrying calibration weights. The typed model attains F1 0.908 against human labels; one frontier LLM gains 0.059 and the other is statistically indistinguishable. Neither the typed model (slope 1.63) nor the more accurate frontier arm (2.55) is calibrated, while the second frontier arm is (0.98), so calibration is model-specific; two-parameter recalibration cuts calibration error 3.3-fold out of fold. Kappa against coded fields understates fidelity to narratives by a median of 0.26. The paper derives a resolution-floor bound for two-decimal probability grids and reframes review budgets over flagged records (held-out 0.8% of flags at 90% precision). Adding narrative variables raises injury and fatal crashes attributed to nine factors by 10,747 per year. Main limitations: rare variables rest on as few as 5 positive labels, unclear judgments were excluded rather than adjudicated, frontier costs were not metered, and both corpus and model are closed.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A typed non-generative model codes 195,857 Texas crash narratives at a measured $0.154 per thousand (~$774 for the 5.02M corpus), with cost governed by schema size rather than narrative length.
- Against 2,416 blinded human judgments the typed model reaches F1 0.908 (weighted kappa 0.906); one frontier LLM is 0.059 better and another is statistically indistinguishable from it.
- Calibration is a property of the individual model rather than the paradigm: the typed model (slope 1.63) and the more accurate frontier arm (2.55) are uncalibrated while the second frontier arm is calibrated (slope 0.98), and out-of-fold recalibration cuts calibration error 3.3-fold.
- Agreement with coded CRIS fields understates fidelity to the narrative by a median 0.26 in kappa because coded-field omission is one-sided.
- A review budget defined over flagged records at a target precision (held-out 0.8% of flags at 90% precision) is the operationally meaningful quantity, and adding the calibrated variables raises injury and fatal crashes attributed to nine factors by 10,747 per year.
Strengths
Population-scale execution with measured, not projected, cost
Stage 1 screened 499,500 narratives for $25.23; Stage 2 coded 195,857 for $30.22 ($0.154 per 1,000; $774 projected for the 5.02M corpus), with 0 failed calls and a fitted token model (α=171, β=122 tokens/question, b=0.236/char, R²=0.987) showing schema size dominates narrative length — reversing the premise of two-stage screening designs (break-even q*=0.67).
Dual-reference calibration audit with survey-grade estimation
Probabilities are scored against coded CRIS fields (agreement only) and 2,416 blinded human judgments sampled by probability cells with Deville–Särndal calibration weights (span ×189, design effect 1.95), using exact grid ECE, calibration slopes, Spiegelhalter tests, cluster bootstrap, and cross-fitted precision thresholds over 100–200 splits.
Honest, quantified failure reporting
Two genuinely novel framings with operational payoff
Actionable cross-model finding
Weaknesses
Severe label scarcity exactly where the model fails
Disputes excluded rather than adjudicated
Frontier comparison is confounded on cost and compute
Lemma 2 rests on an unverified and empirically violated assumption
Closed model, restricted corpus, perishable audit
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether a state safety office can convert ~557,564 annual police crash narratives into analyst-defined variables without unaffordable frontier-model costs or unverifiable generated text. Coding is formulated as gated, typed decisions answered by Jev 1.13, a commercial System One model that returns probabilities on a fixed option grid with no text output. Four claims are advanced: (1) population-scale coding at measured cost ($0.154 per 1,000 narratives; $774 projected for the 5.02M corpus); (2) an independent calibration audit against coded CRIS fields and 2,416 blinded human judgments; (3) a review budget defined over flagged records (held-out 0.8% of flags at 90% precision); (4) a downstream shift — +10,747 injury and fatal crashes per year attributed to nine factors once narrative variables join the coded fields. The claims are carefully scoped to what the narrative states, never what occurred, and the paper is unusually explicit about what it does not measure (frontier costs, cross-state transfer).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Jev model in this crash narrative paper?
Jev (identifier jev-1.13.0, from vendor TypeSafe) is a commercial System One typed-decision model: it takes a state object plus typed questions and returns, per question, an answer from a fixed option set together with a probability, without generating text. This paper is an independent academic audit of that model on Texas crash narratives, not a vendor publication.
How accurate is the model at reading police crash narratives?
Against 2,416 blinded human judgments the typed model reached F1 0.908 and weighted kappa 0.906. For comparison, a frontier LLM scored 0.967 and another 0.885 on the same records. Four variables failed: unbelted restraint, intentional acts, wrong-way travel and vehicle defect.
What does it mean that the probabilities are not calibrated, and why does it matter?
Calibrated means that among records given probability p, about a fraction p are truly positive. Here the model's mean probability (4.8%) over-stated the true prevalence (2.7%), so summing its probabilities over-counts events. Ranking records for review is unaffected, and a two-parameter recalibration fitted on roughly 2,400 human labels reduced the calibration error 3.3-fold out of sample.
Can other states replicate this pipeline for their own crash data?
Not directly: the Texas corpus is restricted by a TxDOT data agreement and Jev is a closed commercial model. The authors release the question schema, the runner and the metric code with unit tests, estimate full-corpus coding at roughly $774, and advise agencies to refit the recalibration map on their own human-labeled narratives and rebuild the coded-field mapping from their own data dictionary.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extraction contains two truncation markers covering parts of Sections 3.3–3.4, the middle of 4.5 and the start of Section 5; figures are present only as axis-text, so figure-level claims rest on the in-text numbers.
- Appendix A (the full question schema) and Appendix B are partially truncated, so criteria completeness and the exact coded-field mapping could not be fully verified.
- Vendor and product claims (Jev, TypeSafe pricing, System One framing) and the frontier model names could not be independently checked offline.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.