Enhancing Extubation Failure Prediction with LLM-Derived Features from Respiratory Therapy Clinical Notes
Izzy Chaiken, Aditya Khowal, Neha A. Sathe, Mark M. Wurfel, Lucy Lu Wang
cs.CL · cs.LG · physics.soc-ph
Quick verdict
Single-center study: LLM-extracted RT-note features add ~+2 AUROC to extubation-failure prediction, but gains lack significance tests and fade temporally.
Summary
The paper builds a single-center EHR cohort of 3,243 adult invasive-ventilation episodes (>=24 h IMV, 7-day failure window, 19.95% extubation failure) from UW Medicine (2021-2023) and augments structured-data EF prediction with features extracted by few-shot prompting of Llama-3-8B-Instruct from free-text respiratory therapy (RT) notes. Fifteen features in five categories (sputum, cough, suctioning, cuff leak, breath sounds) were defined with pulmonologist input; extraction was validated on 200 manually annotated held-out notes (most Macro-F1 >0.80; weak features dropped). Logistic regression with note features raises AUROC from 0.729 to 0.749 (0.733 to 0.752 with medication features) and by 5.1 points among patients with notes; gradient boosting overfits, and survival models add no benefit. Varying minimum IMV duration from 1 h to 24 h moves AUROC 0.796 to 0.752 and positive F1 0.390 to 0.467, supporting the argument that differing cohort definitions make prior EF studies incomparable; the EF window (12 h to 14 d) barely matters. The model beats the institutional checklist CDSS (precision 0.34 vs 0.21). Main limitations: single center, no external validation, no significance testing of the AUROC gains, unquantified extraction-error propagation, and RT-feature benefits vanishing on the 2023 temporal split. Code is public; the data cannot be shared (PII).
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Few-shot Llama-3-8B-Instruct reliably classifies EF-relevant features (sputum, cough, breath sounds, cuff leak) in RT notes, with most Macro-F1 above 0.80 on 200 held-out notes.
- Adding LLM-extracted RT-note features to structured EHR data improves EF prediction: AUROC 0.733 to 0.752 overall and +5.1 points among patients with documented notes.
- Logistic regression outperforms gradient boosting and survival models for this binary EF task, contrary to prior EF-ML literature, which the authors attribute to overfitting and distribution shift.
- Minimum-IMV-duration inclusion criteria systematically drive performance (AUROC 0.796 to 0.752 from 1 h to 24 h), making models trained under different cohort definitions incomparable.
- The LR model with note features outperforms the institution's existing checklist CDSS (PPV 0.34 vs 0.21; 59/81 vs 40/81 EF cases flagged), though the comparison is confounded by treatment effects.
Strengths
Novel, clinically grounded note type and schema
RT notes are absent from public EHR datasets like MIMIC-IV (footnote 1, Sec. 1). The 15-feature schema across 5 categories (Table 2) was derived from literature plus two practicing critical-care pulmonologists and targets established EF risk factors such as cough strength and sputum quantity (Sec. 4).
Disciplined annotation and extraction validation
400 notes labeled by two authors to consensus, then reviewed by two pulmonologists; prompt engineering confined to the 200 training notes and metrics reported on 200 held-out test notes (Sec. 4). Features with Macro-F1 below 0.8 or high correlation (e.g., spontaneous cough 0.53, suctioning) were excluded downstream rather than silently kept.
Cohort-definition sensitivity analysis is a genuine contribution
Unusual breadth of robustness checks
Honest negative results and clinically translated effects
Weaknesses
Headline improvement is never significance-tested
The RT-note gain does not survive temporal validation
LLM extraction error propagation is unquantified
Endpoint and exclusion choices are under-analyzed
Modest absolute performance and no embedding baseline
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is extubation failure and how is it defined in this study?
Extubation failure (EF) is when a patient dies or must be put back on invasive mechanical ventilation within a set window after a planned removal of the breathing tube. This study defines EF as death or reintubation within 7 days of extubation, among patients ventilated for at least 24 hours; 647 of 3,243 patients (19.95%) failed, 618 of them via reintubation.
How much do the LLM-extracted note features improve extubation-failure prediction?
On the held-out test set, adding respiratory-therapy-note features raised AUROC from 0.729 to 0.749 for logistic regression (0.733 to 0.752 when medication features are also included) — a gain of roughly 2 points — and by 5.1 points among patients who had notes in the 12 hours before extubation. Positive-class precision rose from about 0.32-0.33 to 0.34.
Which LLM was used to read the clinical notes and how accurate is it?
The authors prompted Meta-Llama-3-8B-Instruct with few-shot examples (temperature 0.01) to classify 15 features such as sputum quantity and cough strength. On 200 manually annotated held-out notes, Macro-F1 was mostly above 0.80, ranging from 0.53 (spontaneous cough) to 0.99 (thick sputum); weaker features were excluded from the downstream prediction models.
Is the code or data for this paper available?
Reproduction code is publicly available on GitHub (larchlab/extubation-failure). The EHR dataset from University of Washington Medicine contains personally identifiable information and is not publicly available, so the downstream experiments cannot be fully replicated outside the institution.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (confusion matrices, AUROC/F1 curves, calibration and DCA plots) appear only as captions in the extracted text; visual evidence could not be verified.
- Table 4's layout was garbled by text extraction; metric-to-model mappings were reconstructed from surrounding prose and should be checked against the published table.
- The supplemental prompts and GitHub repository contents were not inspected; the reproducibility assessment relies on the paper's availability statements.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.