Emergency Department Revisit Quality Review Screening: Exploring Human Decision-Making and Artificial Intelligence Support
Jonathan A. Handler, Marlene I. Robles-Granda, Jacob E. Mefford, Jeremy S. McGarvey, Gregory S. Podolej, Colleen J. Klein, Matthew D. Dalstrom, William F. Bond
cs.CY · cs.AI
Quick verdict
Study finds GPT-4 over-flags ED revisits (94% yes) but a diagnosis-pair knowledge-graph screen shows 83-100% PPV; promising yet tiny, in-sample, unvalidated.
Summary
This exploratory retrospective study examines whether large language models can help screen emergency department (ED) revisit pairs for quality review. From a US multihospital health system's 2022 encounters, 99 randomly selected diagnosis pairs (index visit plus a revisit within 1-14 days) were assessed — using only the primary diagnoses — by 2-3 clinicians and GPT-4 on diagnosis membership, medical gravity, differential inclusion, complication inclusion, and whether further review was warranted. GPT-4, with minimal prompt engineering, judged 93.9% of pairs warranting follow-up versus 7.1-21.2% for clinicians, correlating poorly. Mixed-effects logistic regression (random intercepts for rater and pair) found revisit medical gravity consistently associated with the target (OR 1.07-1.08, p<0.001); the differential/complication composite was significant unadjusted (OR 1.01, p=0.038) but not adjusted (p=0.104). Informed by these results, the authors derived a knowledge-graph algorithm (KGA) populated by GPT-4o mini and filtered by a Medical Gravity Index; on the same 99 pairs it achieved 83% PPV at cutoff ≥90 (5/6) and 100% at ≥100 (4/4), with only 14-18% sensitivity, and most true positives were 6-7 day revisits that standard 48-72-hour screening misses. Key limitations: tiny sample, derivation and evaluation on the same data, no PPV confidence intervals, subjective ground truth, and a single health system; external validation is required.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- With minimal prompt engineering, GPT-4 correlated poorly with clinician raters, flagging 93.9% of 99 ED revisit diagnosis pairs for follow-up versus 7.1-21.2% for clinicians.
- Among clinician raters, revisit medical gravity was consistently and significantly associated with judging a pair warranting review (OR 1.07-1.08, p<0.001), while the differential/complication composite was significant only unadjusted.
- A knowledge-graph algorithm (KGA) populated by GPT-4o mini achieved 83-100% PPV on the same 99 pairs at MGI cutoffs ≥90 and ≥100, with 14-18% sensitivity.
- Most KGA true positives were 6-7 day revisits that standard 48-72-hour screening would miss, suggesting the approach could widen screening scope with limited added reviewer burden, pending validation.
Strengths
Transparent negative result for GPT-4
The 93.9% 'yes' rate (n=93/99) versus clinicians' 7.1-21.2% is reported frankly, with attribution to minimal prompt engineering and isolated per-item prompts (Discussion), a useful caution for LLM triage deployments.
Statistically appropriate clustered design
Mixed-effects logistic regression with random intercepts for rater and study ID (Tables 1-3) correctly handles repeated ratings per pair; revisit MGV is consistently significant across stages (OR 1.07-1.08, p<0.001).
Novel, implementable screening architecture
Reframed metric choice with honest sensitivity disclosure
Weaknesses
In-sample KGA evaluation
Noisy, lenient ground truth
Under-powered LLM baseline
Small, borderline statistics
Thin clinical inputs and unquantified workload benefit
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: ED Revisit Quality Screening with Human and AI Raters
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What did this study find about using GPT-4 to screen emergency department return visits?
In this exploratory study of 99 diagnosis pairs, GPT-4 with minimal prompt engineering judged 93.9% of pairs as warranting follow-up, whereas clinician raters judged only 7.1-21.2% — a poor correlation that the authors attribute to under-engineered, isolated prompts rather than proof that LLMs cannot help.
What is the 'KGA' algorithm and how accurate was it?
KGA is a screening algorithm using a knowledge graph of diagnosis-pair relationships (differential diagnosis, complications), populated by GPT-4o mini via open-source software, then filtered by a Medical Gravity Index cutoff. On the same 99 pairs it achieved 83% positive predictive value at cutoff ≥90 (5 of 6 flagged correct) and 100% at ≥100 (4 of 4), with only 14-18% sensitivity.
Why do the authors argue precision matters more than sensitivity here?
Standard practice reviews only revisits within 48-72 hours, so its sensitivity for later revisits is effectively zero. The authors argue a high-precision screener that flags a few concerning later revisits (most true positives here were 6-7 day revisits) adds value without overwhelming chart reviewers.
Can these results be applied at other hospitals?
Not yet. The study is explicitly preliminary: a single health system, primary diagnoses only, a subjective clinician 'target' standard, and derivation and evaluation on the same small sample. The authors call for validation before any implementation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's algorithm diagram is represented only by its caption and in-text description in the extracted text; the flowchart's visual detail was not available for review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.