A Living Benchmark for Information Retrieval from Electronic Health Records
Jordan L. Cahoon, Chloe O. Stanwyck, Sulaiman Somani, Philip Chung, Kevin R Keet, Kameron C. Black, Andrea T. Fisher, Sarita Khemani, et al.
cs.AI
Quick verdict
Living EHR retrieval benchmark (BRIE, 508 clinician-validated Qs) from an auto-generator; solid design, yet judge circularity and single-site data temper it.
Summary
The paper introduces BRIE, a 508-question benchmark for clinical information retrieval over longitudinal electronic health records, produced by an LLM-based generator rather than manual curation. Questions are grounded in each patient's admission History & Physical note but must be answerable solely from prior notes, via a two-stage pipeline (fact extraction, then QA generation with Gemini Pro 2.5) and a taxonomy of reasoning depth (276 single-hop, 232 multi-hop), clinical topic, and temporality. Fifteen physicians reviewed pairs on 14 criteria (83.7% clinically relevant, 97.8% chart-consistent, 73.3% naturally phrased); adjudication and filtering yielded the final 508. Evaluating nine models under five inference configurations (38 combinations), the authors find hallucination rare (~99% of answer facts grounded) but omissions frequent, with fact recall dropping 0.10-0.20 on multi-hop questions and 0.16-0.25 when evidence lies beyond 180K tokens for long-context configurations. Scoring against multiple clinician-validated interpretations (87 of 100 questions) raises recall by 0.12 and precision by 0.14, showing single-reference evaluation underestimates performance. Regenerating the benchmark from 2026 admissions (n=1,000) preserves difficulty without clinician review. Limitations: single institution and 68 patients, Gemini generation/judging circularity, agentic testing on only two models, and no clinician validation of regenerated cohorts.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An LLM-based generator, validated once by clinicians, can automatically produce chart-grounded retrieval QA pairs of high quality (83.7% clinically relevant, 97.8% chart-consistent).
- Across nine models and five inference configurations, hallucination is rare (~99% of answer facts grounded) but omission of clinically important facts is frequent, especially for multi-hop questions requiring synthesis across encounters.
- Single-reference evaluation systematically underestimates retrieval performance: scoring against the best of several clinician-validated interpretations raises fact recall by 0.12 and precision by 0.14.
- The generator can refresh the benchmark on new admissions (BRIEnew, n=1,000) while preserving overall difficulty and characteristic failure modes without clinician review.
Strengths
Leakage-controlled generator design
Two-stage generation forces answers to be supported exclusively by facts extracted from pre-admission notes while the H&P supplies only question context; the filtering prompt excludes questions answerable from the H&P, and clinicians scored a dedicated leakage criterion (Online Methods Sections 2-3).
Unusually rigorous validation for a synthetic benchmark
Two physicians per pair across 14 criteria, third-physician adjudication, a fully reported filtering funnel (Supp Table 4: 750 → 675 → 508), a 50-entry review of answer-level disagreements, and CI-annotated validation rates (Figure 1c).
Broad, costed evaluation matrix
Quantified single-reference bias
Mechanistically coherent failure-mode decomposition
Weaknesses
LLM-judge circularity with a permissive entailment rubric
The 'living benchmark' claim is validated distributionally, not qualitatively
Single institution, small patient cohort
Internal numeric inconsistencies
Agentic conclusions rest on two models from one vendor
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
BRIE addresses a real gap: EHR-embedded chart-review assistants lack evaluation data that keeps pace with model churn. The authors make three claims: (1) an LLM generator, validated once by clinicians, produces chart-grounded QA pairs at scale (750 annotated → 675 generated → 508 curated); (2) nine frontier/open models across five inference configurations (38 combinations) rarely hallucinate (~99% of answer facts grounded, Supp Table 8) but systematically omit clinically important facts, especially multi-hop; (3) the generator enables living-benchmark properties — multiple valid reference answers and contamination-resistant regeneration. Claims 1-2 are well supported; claim 3 is only partially validated.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is BRIE?
BRIE (Benchmark for Retrieving Information in EHRs) is a 508-question evaluation dataset for testing how well LLM systems retrieve and synthesize information from longitudinal electronic health records. Its questions and reference answers were generated automatically by an LLM pipeline from de-identified Stanford Health Care notes and validated by clinicians.
What are the main findings of the BRIE paper?
Across nine LLMs and five inference strategies, hallucination was rare (about 99% of answer facts were grounded in the chart) but omission of clinically important information was common, especially for questions requiring synthesis across multiple encounters (fact recall dropping roughly 0.10-0.20). Scoring against multiple valid clinician-verified interpretations raised recall by 0.12 and precision by 0.14, showing that single-reference evaluation underestimates model capability.
Is the BRIE dataset publicly available?
The extracted paper text contains no explicit public release statement. BRIE is built from de-identified Stanford clinical notes, so any access would likely be governed by data-use agreements rather than open download.
How does BRIE avoid benchmark contamination?
Because the question generator itself was clinician-validated, it can be rerun on newly admitted patients to produce fresh questions. The authors regenerated a 1,000-question cohort from 2026 admissions and showed that difficulty and characteristic failure modes were preserved without any new clinician review, though the regenerated items themselves were not clinician-audited.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Large portions of the Results and Online Methods were omitted in extraction (e.g., the 'State-of-the-art models' section with headline per-model aggregate recall/precision); findings cited here come only from the surviving text.
- Figures are described only via captions; kappa values, distributions, and significance markers in Figures 1-6 and Supplementary Figures could not be analyzed.
- No code/data availability statement was visible in the extracted text; codeAvailable=false reflects absence of evidence, not a confirmed negative.
- Model names/dates (GPT-5.4, Claude Opus 4.7, 2026 admissions) are as printed in the provided text and were not independently verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.