Patient-Reported Survey Data Improve Prediction of Opioid Use Disorder
Xiyue Jiang, Zihan Ding, Grace Han, Yinan Liu, Richard N. Rosenthal, Fusheng Wang
cs.LG
Quick verdict
Claims survey data improve EHR-based OUD prediction across all 24 model-window tests; gains are small and may partly reflect informative survey missingness.
Summary
The paper tests whether All of Us participant surveys add predictive value beyond EHR data for identifying a first recorded opioid use disorder (OUD) diagnosis among 267,747 patients with documented opioid exposure (15,287 cases). Eight models—logistic regression, random forest, XGBoost, LightGBM, MLP, LSTM, GRU, and Transformer—are compared under EHR-only and EHR+survey feature sets across 6-, 12-, and 24-month look-back windows. Survey augmentation improved PR-AUC in all 24 model-window combinations, by 0.0087–0.0505; the best model (LightGBM, 24 months) rose from 0.6219 to 0.6603. Permutation importance ranked survey features second among information domains at 24 months in both LightGBM and GRU, behind laboratory measurements. Key limitations, mostly acknowledged by the authors: OUD labels come from diagnostic codes; cases and controls use different index dates; survey coverage differs sharply by outcome status (21.7% vs 60.7% at 24 months), so gains may partly stem from informative missingness rather than response content; no calibration, subgroup, or external validation is reported; and no confidence intervals or significance tests accompany the reported gains. Code is public, but the data require controlled All of Us access.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Adding All of Us survey features to EHR features improves test-set PR-AUC in all 24 model-window combinations, by absolute gains of 0.0087–0.0505, with the best model (LightGBM, 24 months) improving from 0.6219 to 0.6603.
- Permutation importance ranks survey features as the second most important information domain at 24 months in both LightGBM (mean PR-AUC decrease 0.172) and GRU (0.234), behind laboratory measurements.
- Survey availability differs markedly by outcome status (24-month coverage 21.7% OUD-positive vs 60.7% OUD-negative), so part of the observed gain may arise from availability patterns rather than response content.
- Gradient-boosting models outperform LSTM, GRU, and Transformer sequence models, suggesting summary features already capture most useful longitudinal information.
Strengths
Exhaustive paired comparison design
Eight architectures (logistic regression, random forest, XGBoost, LightGBM, MLP, LSTM, GRU, Transformer) × three look-back windows × two feature sets, with identical patient splits shared between EHR-only and EHR+survey models (Methods, Cohort splitting) — enabling the clean paired result that PR-AUC improved in all 24 combinations (Table 5).
Transparent treatment of the survey-availability problem
Table 3 quantifies coverage by OUD status (6→24 months: 9.5%→21.7% OUD-positive vs 21.6%→60.7% OUD-negative, plus questions-answered and days-to-index), and the Discussion explicitly concedes that 'some of the observed gain may therefore arise from patterns of survey availability' — unusually candid.
Leakage-aware preprocessing
Sensible metric and imbalance decisions with evidence
Complete hyperparameter and architecture reporting
Weaknesses
No uncertainty quantification for headline gains
Availability-vs-content confound left unresolved
Differential index-date design unquantified
Sequential models structurally handicapped
Discrimination-only evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed question: do participant-reported survey data add predictive value **beyond structured EHR data** for identifying a *first recorded* OUD diagnosis among opioid-exposed patients (ATC N02A) in All of Us (n = 267,747; 15,287 cases, ~1:16.5)? The headline claim is that survey augmentation improved test-set PR-AUC in **all 24 model–window combinations** by 0.0087–0.0505 (Table 5), best at LightGBM/24m (0.6219 → 0.6603). Supporting claims: survey is the second-ranked information domain at 24 months in LightGBM (0.172 PR-AUC decrease) and GRU (0.234), behind laboratory measurements; and survey availability itself differs by outcome (Table 3: 21.7% vs 60.7% coverage at 24m). Scope discipline is a genuine strength — the outcome is explicitly 'first recorded diagnosis,' not incident disorder, and the abstract does not claim causal or clinical deployment readiness.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How much did survey data improve opioid use disorder prediction over EHR data alone?
Across all 24 model-window combinations tested, PR-AUC increased by 0.0087 to 0.0505 when survey features were added. The best configuration — LightGBM with a 24-month look-back window — improved from 0.6219 to 0.6603.ROC-AUC also improved in every combination.
What data and population did the study use?
267,747 participants in the NIH All of Us Research Program with at least one documented opioid medication exposure (ATC code N02A), including 15,287 participants with a recorded OUD diagnosis (ICD-9 304.00–304.03 or ICD-10 F11). Features came from electronic health records plus four All of Us surveys: The Basics, Lifestyle, Overall Health, and Social Determinants of Health.
What is the main caveat about the survey-data benefit in this study?
Survey completion was much lower among people who later developed OUD (21.7% vs 60.7% at 24 months), so a model can partially learn who answers surveys rather than what they answer. The authors acknowledge this and defer an ablation separating availability from response content to future work, meaning part of the improvement may reflect selection effects.
Is the code or data for this study publicly available?
The analysis code is publicly available on GitHub (StonyBrookDB/AllOfUsOUDPrediction). The underlying All of Us data are controlled-access: researchers must be approved through the All of Us Researcher Workbench to obtain the same cohort.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2's plotted bars and error bars are only partially recoverable because the figure's inline axis/tick text was extracted mid-paragraph; importance magnitudes cited here come from the fully legible body text (LightGBM 0.084/0.156/0.172; GRU 0.046/0.073/0.234), not from reading the figure itself.
- Table 1/4/5 cell values were extracted as run-on text; individual numbers are legible and internally consistent (e.g., LightGBM 24m 0.6603 − 0.6219 = 0.0384 matches Table 5), but column pairing required careful reading, so any single-cell citation carries minor misalignment risk.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.