ObGynLongBench: Revealing the Evidence-to-EHR Gap in Longitudinal EHR Decision-Making
Jun Xiang, Zhijie Bao, Rong Hu, Kaizhou Qin, Wei Chen, Zhongyu Wei
cs.CL · cs.AI
Quick verdict
ObGynLongBench: LLM accuracy drops when evidence must be mined from EHRs across 17 models; audited benchmark, but gold answers come from LLM pipelines.
Summary
The paper introduces ObGynLongBench, a benchmark of 1,500 four-option multiple-choice clinical decision points instantiated on longitudinal EHR histories of 976 pregnant patients from a single Chinese tertiary hospital. Clinical rules mined from obstetric textbooks and guidelines (via GPT-5.3-Codex) are matched to patient histories, and each case is anchored to a pre-decision cutoff with leakage scrubbing to prevent future information exposure. Each case is tested under three input scopes — Evidence-only, Visit-level, and History-level EHR — across 17 LLMs, quantifying an 'Evidence-to-EHR Gap': the best model falls from 79.1% (Gemini-3-Flash, evidence given) to 68.7% (full history), with drops of 7.6–22.7pp per model; medical LLMs fare worse, MedGemma-1.5-4B collapsing to 41.3%. Accuracy declines with evidence complexity (L1 61.0% → L3 54.2%) and input length, and later decision points in the same patient are harder, with early errors associated with later errors. Among access strategies, a search agent scores best (64.2% vs Direct 59.6%) while Static RAG nearly matches Direct at roughly 1/7 the tokens. A three-clinician blind audit of 100 sampled cases estimated ~1% clearly incorrect gold answers. Main limitations: single-center Chinese-language records, MCQ-only evaluation, LLM-driven construction with only a 100-case human audit, thinking mode disabled for open-source models but default reasoning enabled for commercial APIs, and no clinician baseline.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A substantial Evidence-to-EHR Gap exists: models strong with curated evidence (best 79.1%) drop substantially when evidence must be extracted from same-day records or full pre-decision histories (best 68.7%), with per-model drops of 7.6–22.7pp.
- The gap is driven by evidence utilization: accuracy falls monotonically with evidence complexity (L1 61.0% → L3 54.2%) and pre-decision input length, and LLM-judged evidence utilization declines with length.
- Errors are associated within patients: later decision points are harder for all 17 models (Table 7), and failing an earlier decision point predicts failing a later one from the same history (Table 3).
- A medically motivated active-search agent is the best EHR access strategy (64.2% vs Direct 59.6%), while Static RAG nearly matches Direct access at roughly 1/7 the token cost.
Strengths
Controlled within-case three-scope design
Every case is answered at the identical decision point under Evidence-only, Visit-level, and History-level inputs (Section 2.1, Table 2), so the Evidence-to-EHR Gap is measured per model with patient, gold answer, and question held fixed — a genuinely clean control absent from MedAlign/MIMIC-Instr-style evaluations.
Disciplined anti-leakage construction
Appendix A.2 and the grounding-verification prompt (Figure 13) document explicit cutoff selection, automatic removal of same-day diagnoses, enumerated scrub rules, and a fail-closed rule (grounded_pass=false when no cutoff satisfies answerability without leakage).
Independent clinician audit with fit-for-purpose statistics
Above-average statistical hygiene
Actionable access-strategy findings with cost accounting
Weaknesses
Gold answers and questions are LLM-constructed with a thin human audit
Asymmetric inference settings confound cross-family comparisons
Strategy comparison lacks generality and budget parity
Rule-branch locking induces selection bias
Patient-level 'prediction' claim is partially confounded
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ObGynLongBench targets a concrete weakness of medical LLM evaluation: static QA with pre-packaged evidence. The authors build 1,500 four-option MCQ decision points from 976 real pregnancy EHR histories (single Chinese tertiary center; median history 184 days; mean pre-decision input 30.8K tokens, σ=19.8K), each grounded in a rule mined from ObGyn textbooks/guidelines and anchored to a pre-decision cutoff. The central claim is the 'Evidence-to-EHR Gap': the same question answered with curated evidence (Evidence-only), same-day records (Visit), or the full pre-decision history (History) yields large accuracy differences, isolating evidence retrieval and utilization as the bottleneck. Secondary claims: performance degrades with evidence complexity (L1→L3) and input length; errors are correlated within patients; an active-search agent is the best access strategy.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Evidence-to-EHR Gap reported in ObGynLongBench?
It is the accuracy drop that occurs when an LLM must find and integrate decision-relevant evidence itself from electronic health records instead of having that evidence provided in the question. On this benchmark the best model (Gemini-3-Flash) scores 79.1% with evidence provided but only 68.7% when answering from the full pre-decision EHR history, and individual models drop by 7.6 to 22.7 percentage points.
Is the ObGynLongBench dataset publicly available?
The benchmark cases and evaluation code are released on GitHub (evaluation code under MIT license, with all prompts reproduced in the paper's appendix). The underlying real hospital EHRs are de-identified but sensitive single-center Chinese clinical records, so raw record redistribution is restricted; the paper does not state release terms for the derived cases themselves.
Which LLMs were evaluated on ObGynLongBench?
Seventeen models in three groups: commercial flash-tier models (Claude-Haiku-4.5, GPT-5.4-mini, Gemini-3-Flash, DeepSeek-V4-Flash), long-context open-source models (Qwen3-VL 4B/8B, Qwen3.5 4B/9B/35B-A3B, Gemma-4-E4B/26B-A4B, Phi-4-mini), and medical LLMs (HealthGPT-Pro 4B/8B, Hulu-Med-30A3, Lingshu-7B, MedGemma-1.5-4B).
How reliable are the benchmark's gold answers?
Three obstetrics/gynecology residents who did not build the benchmark independently audited 100 randomly sampled cases against the full pre-decision record: 96 gold answers were judged correct, 1 incorrect, and 3 uncertain (inter-rater agreement Gwet's AC1 = 0.927). The authors extrapolate a benchmark-wide clearly-incorrect rate below 5%, though the audit sample is small and no interval estimate is given.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 6's heatmap is extracted as an unaligned run of numbers without reliable row/column mapping, so chapter-level accuracy judgments in this review rely on the prose of Section C.4 (e.g., high Prenatal Care, low ICP/GBS/Prenatal Screening), not on cell readings.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.