Translation of Black-Box Clinical Prediction Models into Standalone Transparent Nomograms: Temporal External Validation in Heart Transplantation
Henry Pigot, Paulo J. G. Lisboa, Sandra Ortega-Martorell, Ivan Olier, Joseph Mahon, Johan Nilsson
cs.LG · stat.AP
Quick verdict
Paper shows black-box clinical models can become auditable nomograms with preserved discrimination under temporal validation; calibration caveats remain.
Summary
The paper presents PRiSM, a framework that converts trained black-box clinical classifiers for tabular data into standalone transparent nomograms. Functional ANOVA decomposes a source model's input-output behavior into univariate and pairwise partial responses; a LASSO fitted to the observed outcome (not the model's predictions) selects and weights them; an optional Phase II Partial Response Network relearns response shapes within the selected scaffold. Applied to prediction of 1-year mortality after heart transplantation in 50,356 SRTR recipients (training 2000–2016, tuning 2017–2019, external validation 2020–2022) across five source models (IMPACT, logistic regression, MLP, random forest, XGBoost). All five baseline nomograms met a prespecified noninferiority criterion in temporal external validation, retaining over 90% of their source model's discrimination above chance; sparse versions mostly failed, and Phase II refinement recovered the criterion for 2 of 5 families. Translated ML nomograms showed no detectable discrimination difference from de novo GAM and EBM, exceeded NAM, and carried fewer terms than EBM. Known-structure controls behaved as expected. Limitations: modest source discrimination (AUROC 0.619–0.646), single imputation with 57% albumin missingness in the tuning cohort, consistently negative calibration-in-the-large with observed-to-expected ratios below 1, no equivalence margins for comparator comparisons, and US-registry-only data.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- All five baseline PRiSM nomograms met the prespecified noninferiority criterion (retaining over 90% of source-model discrimination above chance) in temporal external validation.
- PRiSM transfers effect shapes from the source model while the observed outcome selects and weights terms, distinguishing it from knowledge distillation.
- The decomposition is exact for known-structure sources: LR yielded only univariate effects and IMPACT recovered its published thresholds.
- Sparse simplification traded discrimination for brevity; Phase II PRN refinement recovered noninferiority for only 2 of 5 source models (LR and XGB), with MLP borderline at −0.0003.
- Translated ML nomograms showed no detectable discrimination difference from de novo GAM and EBM, exceeded NAM, and carried fewer terms than EBM.
Strengths
Prespecified, regulator-anchored primary endpoint
Noninferiority uses the retention contrast T = AUROC_nomogram − 0.90×AUROC_source − 0.05 (an FDA M2-style percent-retention margin), tested via paired DeLong covariance, with lambda selection confined to the tuning cohort and the verdict applied only to temporal external validation (Methods, Primary and secondary endpoints).
Known-structure internal controls validate the decomposition
The LR decomposition returned only univariate terms (Fig 3B) and IMPACT recovered its published thresholds (bilirubin 17.1/34.2/68.4 µmol/L; creatinine clearance 30/50 mL/min; Fig 4A), giving a concrete fidelity check most translation methods lack.
Honest reporting of negative and borderline results
Unusual transparency and reproducibility infrastructure
Clinically relevant secondary and subgroup evaluation
Weaknesses
Low discrimination ceiling
Absolute calibration is off and uncorrected
Single imputation under heavy missingness
Comparator claims lack equivalence margins
Parsimony claims are asymmetric
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether a trained clinical black-box model can be translated into a standalone, term-by-term auditable nomogram without sacrificing performance, and whether the result can then be shortened. PRiSM decomposes a source model's input–output behavior by functional ANOVA into univariate and pairwise partial responses, uses a LASSO fitted to the observed outcome (not the model's predictions) to select and weight them, and optionally refits the retained scaffold in a structured Partial Response Network (Phase II). Five source models for 1-year mortality after heart transplantation are tested: the published IMPACT score, logistic regression (LR), an MLP, random forest (RF), and XGBoost. Central claims: (1) all five baseline nomograms met a prespecified noninferiority criterion in temporal external validation; (2) PRiSM transfers effect shapes from the source while the outcome selects and weights, distinguishing it from distillation; (3) ML-derived nomograms show no detectable discrimination difference from de novo GAM/EBM, exceed NAM, and carry fewer terms than EBM; (4) the framework ships as an open-source Python package.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PRiSM?
PRiSM (Partial Responses in Structured Models) is an open-source Python framework that converts a trained black-box tabular classifier into a standalone, transparent nomogram: the model's learned effects are decomposed into one- and two-variable partial-response curves, and a LASSO fitted to the real patient outcome — not the model's own predictions — selects and weights which curves the final nomogram keeps.
Did the converted nomograms lose accuracy?
Not in their baseline form. In temporal external validation on 2020–2022 heart transplants, all five baseline nomograms retained more than 90% of their source model's discrimination above chance, meeting the prespecified noninferiority criterion. Heavily simplified 'sparse' versions mostly failed the criterion, and a refinement step recovered it for 2 of the 5 model families.
What data was used and can I get it?
The study used the US Scientific Registry of Transplant Recipients (SRTR): 50,356 adult heart transplants from 2000 to 2022, split chronologically into training (2000–2016), tuning (2017–2019), and external validation (2020–2022), predicting death within one year of transplant. The registry data cannot be redistributed; access requires an SRTR application, but the open-source code ships with a synthetic dataset for testing.
How well do these models actually predict 1-year mortality after heart transplant?
Modestly. All source models achieved external-validation AUROCs between 0.619 and 0.646, and the nomograms matched them. The authors argue this weak predictive signal is precisely why transparent, auditable models — which clinicians can inspect term by term — matter for such decisions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with explicit '[...sampling gap...]' markers; parts of the Results narrative, some references, and portions of supplemental methods were skipped. Claims above rely only on visible content.
- Figures were assessed from extracted legends and embedded axis text only; the rendered figure images themselves were not visible (the text states Figures 1–4 are separate PDF files).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.