Machine Learning for Pre-Culture ESBL Risk Stratification to Guide Empiric Antibiotic Selection: A 12-Hospital Study of Enterobacteriaceae Cultures
Aravind V. Kuruvikkattil, Lalitha Pranathi Pulavarthy, Rashmita Kudamala, Saptarshi Purkayastha
cs.LG · stat.AP
Quick verdict
Multi-hospital XGBoost claims pre-culture ESBL risk stratification (AUROC 0.76, NPV 95.8%) to spare carbapenems; careful methods, external validation needed.
Summary
The paper develops an XGBoost model that predicts an ESBL-related resistance phenotype at the moment a culture is ordered, using 45 structured EHR features (demographics, Area Deprivation Index, comorbidities, prior antibiotics, prior resistant organisms, procedures, care setting, plus interactions) drawn from 132,955 Enterobacteriaceae cultures (72,217 patients, 14.41% positive) across 12 Mass General Brigham hospitals (ARMD-MGB, 2015-2024). Cultures are partitioned at the patient level with verified zero overlap; the tuned XGBoost reaches AUROC 0.763 and AUPRC 0.435 on the held-out set, with a Brier of 0.098 after Platt recalibration. At a 90%-sensitivity alert threshold, NPV is 95.8%, sparing an estimated 307 unnecessary broad-spectrum courses per 1,000 cultures while missing 14 ESBL-phenotype cases. SHAP ranks prior ESBL history as the dominant predictor; ablating ADI costs only 0.020 AUROC, motivating a deployable 42-feature no-ADI model. Sensitivity analyses cover a strict IDSA ESBL-E label (0.766), specimen-type ablation (+0.001), unweighted training (0.762), organism strata (0.71-0.78), and an ESBL-naive subcohort (0.696). Key limitations: the composite label includes piperacillin-tazobactam resistance and AmpC genera and is not a molecular ESBL diagnosis; carbapenem-sparing safety rests on NPV and decision-curve analysis without patient outcomes; validation is single-system and urine-dominated (89.1%), with no site identifiers and no calendar-time validation due to date-shifting. Code is public; data requires PhysioNet credentialing.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A 45-feature pre-culture XGBoost predicts the ESBL phenotype with AUROC 0.763 and, at a 90%-sensitivity alert threshold, NPV 95.8% (post-test probability 4.2%) on a patient-disjoint test set of 26,369 cultures.
- Prior documented ESBL history is the dominant predictor (mean |SHAP| 0.389), ahead of prior organism burden (0.197) and Area Deprivation Index (0.191).
- Removing all ADI features costs only 0.020 AUROC (0.763 to 0.743), supporting deployment of an equitable 42-feature model without socioeconomic predictors.
- Class-imbalance correction (cost-sensitive weighting or SMOTE) provides no discrimination benefit over unweighted training (0.762 vs 0.760), which also arrives calibrated.
- Discrimination is stable under a strict IDSA ESBL-E label (0.766 retrained), across organism strata (0.71-0.78), and in an ESBL-naive subcohort (0.696).
Strengths
Leakage-aware evaluation design
Patient-level 80/20 partition with verified zero overlap; the paper quantifies the inflation it avoids under culture-level splits (AUROC 0.773 vs 0.760; 0.809 for test cultures whose patient appeared in training vs 0.678 for unseen patients), directly addressing the record-wise-split pitfall it cites (Saeb 2017, Kapoor 2023).
Calibration handled correctly, plus an honest negative control
Platt scaling fitted on patient-grouped out-of-fold training predictions cuts the tuned model's Brier from 0.182 to 0.098 (intercept -0.02, slope 1.02); the paper also shows unweighted XGBoost (0.762) matches cost-sensitive training (0.760) with already-calibrated raw probabilities, citing simulation evidence that imbalance correction buys no discrimination.
Extensive robustness battery
Clustered statistics done right
Deployment-grade transparency
Weaknesses
Safety of carbapenem-sparing never linked to patient outcomes
Composite label stretches beyond ESBL
Single-system, urine-dominated, no external or temporal validation
No simple or clinical baselines
Threshold provenance and marginal tuning gain
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper develops and internally validates an XGBoost model predicting, at the moment a culture order is signed (t0), an ESBL phenotype defined as AST resistance to any of ceftriaxone, ceftazidime, cefepime, or piperacillin-tazobactam. Data: ARMD-MGB (PhysioNet), 132,955 Enterobacteriaceae cultures from 72,217 patients across 12 Mass General Brigham hospitals, 2015-2024; 19,159 positives (14.41%). Three objectives: a clinically actionable NPV for empiric carbapenem-sparing; SHAP quantification of prior colonization versus neighborhood deprivation (ADI); leakage-free patient-level validation. Headline claims: tuned AUROC 0.763 / AUPRC 0.435, NPV 95.8% at a >=90% sensitivity threshold (post-test probability 4.2%; roughly 307 unnecessary broad-spectrum courses spared per 1,000 cultures at the cost of 14 missed); prior ESBL history dominates attribution (mean |SHAP| 0.389); ADI removable at a cost of 0.020 AUROC for equitable deployment. The title's ESBL framing is broader than the label itself, an issue the authors openly discuss.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the model actually predict - true ESBL or something broader?
It predicts a susceptibility phenotype: resistance to any of ceftriaxone, ceftazidime, cefepime, or piperacillin-tazobactam on routine AST. That includes piperacillin-tazobactam resistance and AmpC-mediated resistance in genera like Enterobacter, so it is broader than molecular ESBL. A strict IDSA ESBL-E sensitivity analysis showed nearly identical discrimination (AUROC 0.766 vs 0.763).
How accurate is the model and what would it mean clinically?
On a held-out set of 26,369 cultures it reached AUROC 0.763. At an alert threshold set for 90% sensitivity, negative predictive value was 95.8%: a screen-negative culture carried a 4.2% post-test probability of the resistance phenotype. The authors estimate about 307 unnecessary broad-spectrum courses avoided per 1,000 cultures at the cost of 14 missed cases - but this is model-level retrospective evidence, not a tested clinical workflow, and external validation has not been done.
Can I reproduce this study?
The analysis code is public on GitHub (iupui-soic/AMIA_2026_PriorESBL), but the data (ARMD-MGB on PhysioNet) requires credentialed access under a data use agreement, so full reproduction needs that approval first.
Where does the data come from?
132,955 Enterobacteriaceae cultures from 72,217 patients across 12 Mass General Brigham hospitals (2015-2024), via the PhysioNet ARMD-MGB repository; 14.41% of cultures met the composite ESBL-phenotype definition, and 89.1% were urine specimens.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels (Fig. 1 cohort flowchart, Fig. 2 calibration diagram, Fig. 3 SHAP beeswarm) are captioned in the text but their visual content cannot be verified from the extraction; DCA is auditable only via the point estimates quoted in the Results prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.