A Rubric-Guided Large Language Model Solution for Opioid Use Disorder Computable Phenotyping
Mengxian Lyu, Paredes Pardo, Cheng Peng, Ziyi Chen, Mengyuan Zhang, Jieting Li Lu, Gary M Reisfield, William M Greene, et al.
cs.CL
Quick verdict
Rubric-guided LLM + OPRO prompt tuning tops ML and zero-shot baselines for OUD phenotyping (F1 0.774); promising but single-site, no significance tests.
Summary
The paper presents a rubric-guided large language model (LLM) framework for identifying opioid use disorder (OUD) from electronic health records. Two University of Florida physicians chart-reviewed 300 opioid-enriched patients, yielding 253 definitive labels (68 OUD-positive). The method prompts gpt-oss-120b to judge an 18-item evidence rubric adapted from a physician-developed documentation ruleset, returning binary judgments with verbatim supporting text, then aggregates the 18 indicators via logistic regression into a patient-level OUD probability. Optimization by PROmpting (OPRO) iteratively refines the extraction prompt within outer training folds only. In 5-fold x 3-repeated stratified cross-validation (15 outer test folds), the OPRO-refined pipeline achieved F1 0.774 and AUROC 0.934, versus 0.686/0.890 for the strongest machine-learning baseline (EHR+NLP features) and 0.536/0.805 for zero-shot gpt-oss-120b — relative F1 gains of 12.8% and 44.4%. Versus zero-shot, rubric guidance sharply improved specificity (0.867 vs 0.602) and PPV (0.715 vs 0.415). Key limitations: single-site evaluation with no external validation; differences reported descriptively without confidence intervals or significance tests; 47 UNKNOWN cases excluded; item-level extraction accuracy and evidence faithfulness never validated against chart review; and a Discussion claim that structured EHR data feed the model contradicts the narrative-only Methods. No code or data availability is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A rubric-guided gpt-oss-120b pipeline with logistic regression aggregation achieves the best OUD phenotyping performance on the chart-reviewed cohort (fold-mean F1 0.774, AUROC 0.934), beating EHR+NLP machine learning by 12.8% and zero-shot LLM by 44.4% in relative F1.
- OPRO prompt refinement improves the rubric pipeline from F1 0.755 to 0.774 and AUROC 0.891 to 0.934, with prompt generation and selection confined to outer-training data.
- Rubric guidance corrects zero-shot over-flagging: specificity rises from 0.602 to 0.867 and PPV from 0.415 to 0.715, while sensitivity also increases from 0.775 to 0.858.
- The framework links verbatim supporting text to each item judgment and the final OUD flag, enabling explainable, evidence-based phenotyping.
Strengths
Disciplined leakage control
OPRO prompt search, learner/hyperparameter selection by average precision, Youden threshold estimation, and logistic regression fitting are all confined to outer-training folds; the prompt is frozen before test evaluation (Statistical analysis section).
Well-controlled LLM comparison
Zero-shot and rubric-guided use the same gpt-oss-120b at temperature 0 with a high reasoning profile, so Table 3's specificity (0.602 to 0.867) and PPV (0.415 to 0.715) gains isolate the rubric+aggregation design rather than model choice.
Evidence-linked outputs by design
Transparent gold-standard construction
Clinically grounded rubric
Weaknesses
No inferential statistics
Core novelty unevaluated
Internal inconsistencies
Transferability unknown
No code or data availability
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This preprint (arXiv 2609.05682, cs.CL) addresses opioid use disorder (OUD) computable phenotyping (CP): automatically flagging OUD cases from electronic health records where diagnosis codes are known to undercode (~10% capture among prescribed opioids, per the Introduction citing Ranapurwala et al.). The central claims are: (1) a rubric-guided LLM pipeline — gpt-oss-120b judging an 18-item evidence rubric, aggregated by logistic regression — outperforms a diagnosis-code rule, three ML-based CPs, and a zero-shot LLM, reaching fold-mean F1 0.774 and AUROC 0.934 on 253 chart-reviewed patients; (2) OPRO prompt optimization adds a further gain (F1 0.755 to 0.774; AUROC 0.891 to 0.934); (3) the system is explainable because every met item carries verbatim supporting text. A fourth, inconsistent claim appears in the Discussion: that the CP 'uses both the evidence extracted from the narratives and the patient-level information from structured EHRs.'
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What accuracy did the rubric-guided LLM achieve for identifying opioid use disorder?
On 253 chart-reviewed patients (68 OUD-positive) from UF Health, the OPRO-refined rubric-guided gpt-oss-120b pipeline reached a fold-mean F1 of 0.774 and AUROC of 0.934 across 15 cross-validation test folds, with sensitivity 0.858, specificity 0.867, PPV 0.715, and NPV 0.944 — the best of all methods compared in the paper.
How does the method work?
The LLM reads a patient's selected clinical notes and judges 18 predefined evidence items (e.g., fentanyl-positive urine test, documented heroin use, medication treatment for OUD), returning a met/not-met judgment plus a verbatim supporting quote for each item met. An 18-input logistic regression converts these judgments into a patient-level OUD probability. OPRO, an automatic prompt-optimization method, tunes the extraction prompt using training data only.
Is the code or dataset public?
No code or data availability statement appears in the paper. The chart-reviewed gold standard comes from UF Health electronic health records under IRB approval and is not public; gpt-oss-120b itself is an open-weight model.
How much better is it than just asking the LLM zero-shot?
Substantially on false positives: compared with zero-shot gpt-oss-120b, the rubric-guided version raised specificity from 0.602 to 0.867 and PPV from 0.415 to 0.715 while sensitivity also improved (0.775 to 0.858); F1 rose from 0.536 to 0.774, a 44.4% relative gain as reported.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is present only as a caption; the diagram's visual content is not recoverable from the text extraction.
- Superscript citation numbers can be extraction-sensitive; the reported off-by-one reference numbering should be confirmed against the PDF layout.
- No data or code availability statement appears anywhere in the provided text (verified through the end of the references) — this is a fact about the paper, not an extraction gap.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.