Automated Identification of Complex Percutaneous Coronary Intervention from Cardiac Catheterization Reports Using Large Language Models
Bhatt, N., Warner, F., Miao, J., Thakker, R., Joodi, G., Cantero-Schaffer, P., Huang, C., Krumholz, H., et al.
cardiovascular medicine
Quick verdict
Claims Llama 3.3 70B accurately extracts complex PCI variables from cath notes; credible, but 57.6% PPV and size-confounded model comparisons temper it.
Summary
The study evaluates whether open-weight large language models can automate 'complex PCI' phenotyping from free-text cardiac catheterization reports. A random sample of 1,412 procedure notes (2011-2017) from three Yale New Haven Health hospitals was annotated by four interventional fellows and adjudicated by a board-certified interventional cardiologist. Three locally deployed models - Llama 3.3 70B, Meditron-7B, and BioMistral-7B, prompted identically without fine-tuning - first classified whether a note described PCI, then extracted six variables defining complex PCI (Giustino criteria). Llama 3.3 70B dominated: PCI identification reached 100% sensitivity, 93.8% specificity, 96.4% accuracy (F1 95.9%), while complex PCI classification (590 evaluable reports) achieved 97.7% sensitivity, 80.1% specificity, but only 57.6% PPV (F1 72.5%). Exact-match extraction ranged from 80.7% (lesions treated) to 96.8% (stent count); interpretive variables (lesions, CTO, bifurcation strategy) lagged explicitly documented ones. Both 7B domain models were essentially unusable (Meditron specificity 7.8%; BioMistral sensitivity 1.2%). Site-level heterogeneity was significant for Llama's complex PCI accuracy (p<0.001, I2-style 87%). Limitations include a single health system, notes dated 2011-2017, 26 non-evaluable Llama outputs left unexplained, no inter-rater reliability metrics, and model comparisons that confound scale with domain adaptation.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Llama 3.3 70B identifies PCI in catheterization notes with 100% sensitivity, 93.8% specificity, 96.4% accuracy (F1 95.9%), outperforming Meditron-7B and BioMistral-7B.
- For complex PCI classification among 590 evaluable reports, Llama 3.3 70B achieves 97.7% sensitivity and 80.1% specificity, but PPV is only 57.6% (F1 72.5%).
- Extraction accuracy is highest for explicitly documented numeric variables (stent count 96.8%, total stent length 93.8%) and lower for variables requiring interpretation (lesion count 80.7%, CTO 85.1%).
- Smaller domain-adapted 7B models are not viable for this task, failing in opposite directions (Meditron specificity 7.8%; BioMistral sensitivity 1.2%).
- Performance varies significantly across the three sites for Llama 3.3 70B complex PCI classification despite similar complex-PCI case mix (~21.5-22.2%).
Strengths
Expert-built reference standard on a large annotated cohort
1,412 full catheterization narratives annotated by four interventional cardiology fellows against a predefined schema, then re-reviewed by a board-certified interventional cardiologist (KM) - far larger than the prior comparable PCI-LLM study the authors cite, and grounded in clinically operational Giustino complex-PCI criteria.
Multi-site design with formal heterogeneity quantification
Three hospitals with demonstrably distinct documentation (within-site cosine similarity 0.083-0.389; median note length 495-863 words); site differences tested with Pearson chi-square plus an I2-style statistic (Table 2), and Wilson 95% CIs used throughout for proportions.
Transparent accounting of missing and indeterminate data
Realistic privacy-preserving deployment and reusable artifacts
Actionable, quantified error analysis
Weaknesses
Model comparison confounds scale with domain adaptation
Complex PCI classification is far weaker than the framing suggests
No non-LLM baseline
Annotation reliability never quantified; potential adjudicator circularity
Reporting gaps around evaluable denominators and inference configuration
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How accurate is Llama 3.3 70B at identifying PCI procedures in cardiac catheterization reports?
In this study of 1,412 notes from three Yale New Haven Health hospitals, Llama 3.3 70B identified PCI with 100% sensitivity, 93.8% specificity, 92.1% PPV, 100% NPV, 96.4% accuracy, and an F1 of 95.9%. Its main errors were flagging IVUS/FFR-only procedures as PCI (70% of false positives).
Can LLMs replace manual chart abstraction for complex PCI research?
Partially. The best model extracted explicitly documented variables very well (stent count 96.8%, total stent length 93.8% exact match), but for classifying a case as complex PCI overall, the positive predictive value was only 57.6% - so flagged cases would still need human verification, while negative calls (NPV 99.2%) are more trustworthy.
Which language models were compared and did smaller medical AI models help?
Three open-weight models were compared: general-purpose Llama 3.3 70B versus two medically adapted 7B models, Meditron-7B and BioMistral-7B. The medical 7B models performed far worse - Meditron-7B had 7.8% specificity (labeled almost everything PCI) and BioMistral-7B had 1.2% sensitivity (missed almost all PCI) - suggesting model capacity mattered more than medical pretraining.
Is the code or dataset from this study publicly available?
Evaluation code and prompt templates are available on GitHub (itsjustnilay/complex-pci-llm-evaluation). The annotated catheterization reports themselves are not shared because they contain identifiable clinical data protected by IRB/privacy constraints; all models ran locally to keep notes inside the health system.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (1-4) appear only as captions in the extracted text; error-bar widths, Wilson CI ranges, and per-site bar values could not be visually verified.
- Supplementary materials (Supplementary Table 1 criterion-level metrics, Supplementary Figure 1-2, and the exact prompt template) were not included in the extracted text.
- The abstract in the source metadata is truncated, but the complete abstract appears in the paper body and was used for this review.
- Extracted text contains OCR/license-boilerplate noise; a small number of quoted metrics were cross-checked by re-derivation from the stated confusion-matrix counts.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.