Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark
Christine Park, Valerie Chen, Tim Dettmers
cs.AI · cs.DB
Quick verdict
Open synthetic EHR benchmark with traceable labels and solid validation; realism study is small and LLM-built ground truth softens the verifiability claim.
Summary
The paper introduces Synthetic Hospital, an openly licensed benchmark of 1,268 fully synthetic longitudinal EHR patients (5,602 encounters) built exclusively from public medical-education material. A deterministic five-stage pipeline extracts diagnoses and findings from board-style questions, grounds them to ICD-10-CM, SNOMED CT, and LOINC, clusters questions into multi-encounter patients, and renders records through a FHIR R4-simulated hospital system; benchmark labels are computed from the ontology-grounded knowledge graph before narrative generation, preserving provenance and independence from the rendering LLM (Kimi 2.5). Validation includes a blinded study in which 10 physicians distinguished synthetic from format-harmonized real MIMIC-IV records at 53% accuracy (chance-level), and a pre-registered check showing the graph recovers 93% (111/119) of physician-specified clinical relationships. Across 10 frontier and open models on four chart tasks, no model approaches ceiling: the best severity-weighted F1 for longitudinal problem-list reconstruction is 0.73, matching the mean but not the best (0.89) of seven physicians on a 13-patient subset, and summarization hovers near 0.5 finding-level F1, dominated by omission. A limited agentic study finds multi-turn tool use helps only for patient diagnosis. Main limitations: the case mix is education-derived rather than population-calibrated, the benchmark omits missingness and documentation errors, ground truth rests on LLM extraction without a precision estimate, and both the realism study (100 judgments) and physician comparison (13 patients) are small.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Synthetic Hospital is the first benchmark to combine fully synthetic longitudinal EHRs, ontology-grounded provenance, verifiable ground truth, and unrestricted open access (Table 1, Appendix I).
- Licensed physicians distinguish synthetic records from format-harmonized real MIMIC-IV records at near-chance 53% accuracy.
- The knowledge graph recovers 111/119 (93%) of pre-registered physician-specified clinical relationships.
- Across 10 models and four chart tasks, no model approaches ceiling: best severity-weighted F1 is 0.732, matching the physician mean (0.664) but below the best physician (0.89) on a 13-patient subset.
- Multi-turn agency hurts outside patient diagnosis (loop effect negative in 9/9 model-task pairs) but self-retrieving agents gain +0.14 to +0.34 severity-weighted F1 on patient diagnosis.
Strengths
Provenance-first construction decouples labels from prose
Labels are deterministic functions of the ontology-grounded graph computed before narrative generation (Section 3, Fig. 1); chart-neutral scoring neither credits nor penalizes documented-but-ungraphed diagnoses, and the generator-robustness study (Appendix C.2) shows re-rendering with GPT 5.3 leaves the leaderboard stable (deltas within ±0.05 on 3/4 tasks; GPT drops most on its own rendering, −0.144 retrieval).
Unusually rigorous realism validation
Blinded 10-physician study (Section 4.1) with a carefully documented MIMIC-IV format-conversion pipeline that removes documentation-style confounds: 53% accuracy (95% CI 43–63%, binomial p=0.62), Fleiss' κ=−0.05, no physician above chance after Holm correction.
Pre-registered ground-truth validation with honest gap accounting
Complete evaluation infrastructure beyond a static dataset
Explicit, pre-specified case-mix characterization
Weaknesses
'Verifiable' means traceable, not verified-correct
Headline human comparisons are undersized
Educational case mix limits external validity
Benchmark omits the noise it cites as motivation
Agentic conclusions rest on a single scaffold
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Synthetic Hospital (arXiv 2609.30027) is a benchmark paper: 1,268 fully synthetic longitudinal EHR patients (5,602 encounters; 12,014 scored instances) constructed entirely from public medical-education material and served through a FHIR R4-simulated hospital with role-based access and a 13-tool function-calling API. Its two motivating barriers — real EHRs cannot be openly shared, and their ground truth is only what was documented — are addressed by deriving every label from an ontology-grounded knowledge graph (ICD-10-CM, SNOMED CT, LOINC) before narrative generation. Five headline claims: (1) no prior benchmark combines open data with coverage of longitudinal reasoning, retrieval, problem lists, summarization, and EHR operation (Table 1); (2) physicians distinguish synthetic from format-harmonized real records at 53% (Section 4.1); (3) the graph recovers 111/119 (93%) pre-registered physician relationships (Section 4.2); (4) across 10 models and four tasks nothing approaches ceiling — best severity-weighted F1 0.732 (Kimi 2.5-thinking) vs physician mean 0.664 and best physician 0.89 on a 13-patient subset (Table 3); (5) multi-turn agency helps only patient diagnosis (+0.14 to +0.34; Table 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Synthetic Hospital benchmark?
An open benchmark of 1,268 fully synthetic, longitudinal electronic health record patients (5,602 encounters) built entirely from public medical-education material such as board-style exam questions. Every diagnosis and finding is mapped to standard ontologies (ICD-10-CM, SNOMED CT, LOINC) with provenance back to its source, and records are served through a FHIR R4-based simulated hospital system with no real patient data.
How realistic are the synthetic EHR records?
In a blinded study, 10 licensed physicians tried to distinguish synthetic records from real MIMIC-IV records converted to the same format and were correct only 53% of the time — statistically indistinguishable from the 50% chance level. This shows individual records read plausibly; it does not mean the patient population matches real epidemiology, since the case mix deliberately follows educational material.
How well do AI models perform on Synthetic Hospital?
None of the 10 tested frontier and open models approaches ceiling. The best model reconstructs a patient's longitudinal problem list with a severity-weighted F1 of 0.73, on par with the average of seven physicians but below the best physician (0.89) on a 13-patient subset. When summarizing charts, models miss roughly half of clinically relevant findings — omission, not fabrication, is the dominant failure mode.
Can the benchmark be used to train or fine-tune models?
Yes. Beyond the 200-patient public evaluation split, an 800-patient training split ships with full ground truth and deterministic, verifiable per-instance rewards suitable for reinforcement learning, while a 268-patient held-out split keeps labels accessible only through the authors' scorer to limit contamination.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extracted text omitted several segments (part of Section 3 stage 4, portions of Appendices C.2/E/F and the references); Figures 1–2 and Table 1 check/partial symbols could not be analyzed from text alone.
- Physician-study logistics (IRB status, compensation, physician recruitment) were not visible in the extracted text.
- All assessments are based solely on the provided preprint text; the released repository was not inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.