Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models
Keisuke Masuda, Kazutaka Yatsushiro, Hirohumi Iwamoto, Hirofumi Hirano, Ryosuke Hanaya
cs.CL
Quick verdict
Human-evaluated Japanese stroke LLM benchmark: only 2/18 models pass a safety threshold; valuable, but single-evaluator, single-run design limits conclusions.
Summary
The authors introduce Japanese Stroke LLM Evaluation, a multi-turn conversational benchmark in which the LLM plays the physician while a board-certified neurosurgeon with ten years of clinical experience plays the simulated patient and scores every session. Ten fictional stroke and related cases are each scored out of 100 (50 history-taking, 50 action) under a pre-published deduction rubric; errors that could directly threaten life are pre-defined as critical mistakes, and the safety threshold requires at least 80% overall with zero critical mistakes. Eighteen models — nine cloud-based, nine on-premise — were each evaluated once, in October 2025 or June 2026. Claude Fable 5 led at 87.4% with zero critical mistakes, followed by Claude Opus 4.7 (80.3%, zero); these were the only models to pass. Eleven models committed 17 critical mistakes, nine concentrated in the hyperacute ischemic stroke case, chiefly skipped laboratory checks and use of the Western rather than Japanese t-PA dose. Mean scores rose from 48.7% (October 2025 cohort) to 69.5% (June 2026 cohort), and question count correlated with history-taking score (r = 0.648, p = 0.007, n = 16). Key limitations: a single unblinded evaluator with no inter-rater reliability testing, one run per model, a confounded cross-cohort comparison, mixed web/API execution settings, and only ten cases.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Of 18 models, only Claude Fable 5 (87.4%) and Claude Opus 4.7 (80.3%) met the pre-specified safety threshold of ≥80% overall with zero critical mistakes.
- Eleven models committed 17 critical mistakes, nine concentrated in the hyperacute ischemic stroke case, including four administrations of the Western rather than Japanese t-PA dose.
- Mean overall scores rose roughly 21 percentage points between the October 2025 cohort (48.7%) and June 2026 cohort (69.5%).
- Mean history-taking question count correlates positively with history-taking score (r = 0.648, p = 0.007, n = 16), implicating premature closure in low-scoring models.
- Interactive safety evaluation specific to language, region, and disease domain is necessary before LLMs can support stroke care, because English benchmark performance does not transfer to local practice.
Strengths
Human expert evaluation instead of LLM-as-judge
A board-certified neurosurgeon with 10 years' experience played every simulated patient and scored all 180 model-case sessions against pre-published criteria (Supplementary Table A2), avoiding the self-preference and verbosity biases of judge models the paper cites (§4.1, ref [11]).
Safety-first design with an explicit critical-mistake construct
Case-specific life-threatening errors zero the affected phase and are tracked independently of overall score, cleanly separating competent-but-unsafe models (DeepSeek-V4-Pro: 64.3% overall yet one critical mistake, §3.1/Suppl. B) from threshold passers.
Captures region-specific clinical practice
Contamination-resistant, transparent materials
Quantifies premature closure
Weaknesses
Single unblinded evaluator is also the patient
One run per model undermines the safety conclusions
Confounded generational comparison
Heterogeneous execution conditions
Thin, partially unlabeled statistics
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This benchmark paper introduces **Japanese Stroke LLM Evaluation**, a multi-turn conversational evaluation in which LLMs act as physicians in Japanese-language stroke scenarios. Ten fictional cases (three hemorrhagic presentations, four ischemic strokes, two aneurysm-related, one primary headache; Table 2) are each scored out of 100 (50 history-taking, 50 action) under a pre-published deduction rubric (Supplementary Table A2). Life-threatening errors are pre-defined per case as 'critical mistakes' that zero the affected phase; the pre-specified safety threshold is ≥80% overall plus zero critical mistakes. Central claims: (1) of 18 models (nine cloud, nine on-premise), only Claude Fable 5 (87.4%) and Claude Opus 4.7 (80.3%) — both with zero critical mistakes — meet the threshold; (2) 17 critical mistakes occurred across 11 models, nine in the single hyperacute ischemic stroke case (Case 4), including four models using the Western rather than the Japanese t-PA dose; (3) mean overall score rose from 48.7% (October 2025 cohort) to 69.5% (June 2026 cohort); (4) history-taking question count correlates with history-taking score (r = 0.648, p = 0.007, n = 16).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which AI model performed best in the Japanese stroke LLM benchmark?
Claude Fable 5 scored highest at 87.4% with zero critical mistakes, followed by Claude Opus 4.7 at 80.3% with zero critical mistakes. Out of 18 models tested, only these two met the study's safety threshold: at least 80% overall plus no life-threatening ('critical') errors.
What counts as a critical mistake in this benchmark?
A case-specific action that could directly endanger the patient's life — for example giving the clot-busting drug t-PA without checking lab results or blood glucose, using the Western t-PA dose instead of the Japanese dose, operating before securing the airway, or skipping neck-vessel evaluation. A critical mistake zeroes the score for that phase of the case.
Why does t-PA dosing matter in Japan?
Japan uses a lower t-PA dose (0.6 mg/kg) than most Western countries (0.9 mg/kg). In the benchmark's hyperacute stroke case, four of the 18 models gave the Western dose — an error carrying real bleeding risk that an English benchmark merely translated into Japanese would fail to catch.
Are the benchmark cases and scoring rules public?
Yes. Bilingual case descriptions, simulated-patient response rules, and scoring criteria are available on GitHub (link in the paper). However, the original results depended on one neurosurgeon playing the patient and grading every conversation, so exact replication requires comparable expert involvement.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 (framework diagram, overall performance chart, correlation scatter) were not analyzable from the extracted text.
- Per-case score sheets and conversation transcripts are not included in the text; verification is limited to the aggregate supplementary tables.
- The abstract in the supplied metadata is truncated mid-sentence; conclusions were taken from the full text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.