Automated Chest CT Protocol Selection via Large Language Model Derived Text Embeddings from Imaging Request Text
Zahra Hosseini, Mahan Pouromidi, Farzad Khalvati, Patrik Rogalla
cs.LG
Quick verdict
LLM embeddings pick chest CT protocols at 80% vs 83% for routine assignment (p=0.263); solid large study, but no NLP baseline and equivalence unproven.
Summary
This retrospective single-center study evaluates whether text embeddings from a fine-tuned LLaMA-3.1-70B can automate chest CT protocol selection from free-text imaging requests. The authors concatenated procedure names, clinical indications, and HIS comments from 285,123 requests (2017–2024), normalized abbreviations with CLEVER, and mapped labels onto an 18-class protocol ontology. LoRA-adapted model hidden states were mean-pooled and fed to a class-weighted multinomial logistic regression, all trained locally on one H100 to keep PHI in-house. On 57,024 held-out requests, the pipeline reached 79% accuracy with weighted precision 0.847 and weighted F1 0.81. In a 300-case reader study using four-radiologist consensus as reference, the model scored 80.33% versus 83.33% for routine clinical assignment (McNemar p=0.263); it beat the routine assignment on some classes (e.g., noncontrast chest 89% vs 85%, lung transplant 92% vs 78%) but collapsed on rare ones (PE+R+C 33%, trauma 63%). Assignment entropy was higher for the model (3.747) than for individual radiologists (3.449–3.555). Key limitations: the large test set's ground truth is the routine assignment itself, no bag-of-words or prior CLEVER baseline is reported, the promised rare-class and per-class t-test analyses are absent, and the non-significant p-value is presented as equivalence without a formal equivalence test.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Mean-pooled final-layer hidden states from a locally LoRA-fine-tuned LLaMA-3.1-70B, fed to a class-weighted multinomial logistic regression, assign 18 chest CT protocols at 79% accuracy / weighted F1 0.81 on 57,024 held-out requests.
- Against four-radiologist consensus on 300 cases, the LLM (80.33%) is statistically indistinguishable from routine clinical assignment (83.33%) by McNemar's test (χ²=1.25, p=0.263).
- The LLM distributes protocol assignments more evenly than individual radiologists (entropy 3.747 vs 3.449–3.555), suggesting reduced over-selection of common protocols.
- Fine-tuning entirely within hospital infrastructure (single H100, LoRA, 4-bit quantization) preserves PHI while adapting the model to institutional clinical language, providing a deployable blueprint.
Strengths
Corpus scale and clinical realism
285,123 real RIS/HIS requests over 8 years (2017–2024) with a long-tailed 18-class ontology (Table S2: R 46.3%, R+C 24.3%, LDCT 14.7%) — among the largest chest CT protocoling datasets reported, with an untouched 57,024-case test split.
Privacy-preserving, fully local pipeline with a concrete recipe
LoRA adaptation (r=8, α=32, dropout 0.05, 1500 steps, AdamW-8bit, 4-bit NF4 via Unsloth) and embedding extraction on a single institutional H100; no PHI leaves the hospital — a genuinely reusable deployment template for privacy-restricted jurisdictions.
Unusually candid reader-study reporting
Analysis beyond raw accuracy
Weaknesses
The central hypothesis is never tested against a baseline
Ground-truth circularity on the large test set
'Radiologist-level' claim is calibrated against practice-as-usual, not the experts
Statistical interpretation overshoots the evidence
Unspecified fine-tuning objective and possible shortcut
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Retrospective, REB-approved, single-center study (University Health Network, Toronto) covering 285,123 chest CT imaging requests from 2017–2024, split 80/20 into 228,099 training and 57,024 test cases, labeled over an 18-class protocol ontology (protocols with <100 occurrences excluded; Table S2 shows the extreme skew: R 46.3%, R+C 24.3%, LDCT 14.7%). The system concatenates procedure name, free-text indication, and HIS comments, expands abbreviations via CLEVER, extracts mean-pooled final-layer hidden states from a LoRA-adapted LLaMA-3.1-70B, and trains a class-weighted multinomial logistic regression. The paper claims: (1) 79% accuracy / weighted F1 0.81 on the held-out set (Table 1); (2) radiologist-level performance — 80.33% vs 83.33% for routine clinical assignment against four-reader consensus on 300 cases, McNemar p=0.263 (Tables 2–4); (3) more balanced protocol use than individual readers (entropy 3.747 vs 3.449–3.555, Figure 2); (4) a PHI-preserving local fine-tuning blueprint. The Introduction motivates the work by the failure of keyword/bag-of-words methods and cites the authors' own CLEVER system (ref 14) on the same 300-case set — yet no classical NLP baseline is ever run.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How accurate is the LLM at selecting chest CT protocols?
On 57,024 held-out imaging requests the pipeline reached 79% overall accuracy (weighted precision 0.847, weighted F1 0.81) across 18 protocol classes. On a 300-case expert reader study, it agreed with four-radiologist consensus on 80.33% of cases, versus 83.33% for the protocols actually assigned in routine clinical workflow; the difference was not statistically significant (McNemar p=0.263).
Does the LLM outperform radiologists at protocol selection?
Not overall. Against the same consensus standard, the model scored 80.33% while routine clinical assignment scored 83.33%. The model did beat the routine assignment on some classes — e.g., standard noncontrast chest (89% vs 85%), lung transplant follow-up (92% vs 78%), and PE/CTPA (83% vs 77%) — but was much worse on rare protocols such as PE+R+C (33%) and trauma chest (63%).
Can hospitals run this without sending patient data to the cloud?
That is a core design point of the paper: LLaMA-3.1-70B was fine-tuned locally on a single institutional H100 GPU using LoRA and 4-bit quantization, so no protected health information leaves hospital infrastructure. Note that the paper does not release code, model weights, or data.
What are the main limitations of this study?
It is a single-center retrospective study; the ground truth on the large test set is the routine clinical assignment itself; no traditional NLP baseline (e.g., TF-IDF or the authors' prior CLEVER system) is compared; rare protocols perform poorly; the reader study used a class-balanced 300-case sample; and the 'radiologist-level' conclusion rests on a non-significant p-value without a formal equivalence test.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 are present only as captions in the extracted text; their plotted values (long-tail curve, entropy bars) could not be independently verified, so any result conveyed solely in figure form (e.g., a graphical rare-quartile analysis) would not be visible to this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.