Target leakage, not model class, explains reported accuracy in survey-based cardiovascular screening: a leakage-tiered audit of glass-box and tabular foundation models
Raad Bin Tareaf, Murad Al-Rajab, Samia Loucif, Samer Ellaham, Cedric Schmitz
cs.CL
Quick verdict
Claims BRFSS MI-screening's ~0.89 AUROC stems from two leaked post-diagnostic features, not model class; rigorous multi-axis audit, convincing but BRFSS-only.
Summary
The paper asks whether AUROC ≈0.89 routinely reported for machine-learning cardiovascular screening on the CDC's BRFSS survey reflects learning or target leakage. It benchmarks ten classifiers — logistic regression (with splines), random forest, XGBoost, LightGBM, CatBoost, a multilayer perceptron, the glass-box Explainable Boosting Machine (EBM), and zero-shot tabular foundation models TabPFN v2 and TabICL — on 442,067 respondents of BRFSS 2022 (5.68% prevalent self-reported myocardial infarction), across five nested feature tiers of decreasing leakage risk, then applies frozen models and thresholds to 430,755 respondents of 2023. Removing two post-diagnostic features (prior angina/CHD diagnosis, chest CT) cost every model 0.049–0.051 AUROC, collapsing all ten into a 0.0045-wide band near 0.84. EBM was non-inferior to every alternative within a pre-specified δ=0.005 (paired DeLong, Holm) while scoring 88,413 respondents in 0.1 s versus 1,153.8 s of GPU time for TabICL. A single threshold detected 75.4% of women's versus 89.0% of men's infarctions; shape repair with intercept equalization closed the gap to ~0.010, matching per-group thresholds and reweighing. Marginal conformal coverage ranged 0.94 (women) to 0.82 (adults 60+); Mondrian calibration repaired all strata at the price of deferral or empty prediction sets. Limitations: self-reported prevalent outcome, residual conditioning-on-care in T1, single-country data, and confirmatory tests confined to the seed-0 test partition.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Two post-diagnostic features — prior angina/CHD diagnosis and chest CT — account for ~0.05 of the ~0.89 AUROC reported for BRFSS myocardial-infarction screening, with near-uniform cost (0.049–0.051) across all ten model classes.
- Once leakage-prone features are removed, the glass-box EBM is statistically non-inferior (δ=0.005, paired DeLong, Holm) to tuned gradient boosting and to full-context TabICL, at roughly four orders of magnitude lower scoring cost (0.1 s vs 1,153.8 s for 88,413 respondents).
- A single screening threshold detects 75.4% of women's versus 89.0% of men's infarctions; EBM shape-function repair with intercept equalization closes the TPR gap to ~0.010, statistically matching per-group thresholds and reweighing at matched specificity.
- Marginal split conformal prediction meets its 90% guarantee on average while delivering 0.94 coverage to women and 0.82 to adults 60+; Mondrian calibration repairs every stratum at the price of deferral (0.586→0.712 in 60+) or empty prediction sets (9.5% in ages 18–39).
- Frozen 2022 models and thresholds transport to the full 2023 cycle within 0.002 AUROC (per-model losses 0.0009–0.0016) with sensitivities 0.855–0.862.
Strengths
Clean dose–response demonstration that the 0.89 regime is feature-driven
Removing the two post-diagnostic markers (prior angina/CHD diagnosis, chest CT) cost 0.049–0.051 AUROC in every one of ten model classes, collapsing the field into a 0.0045-wide band (0.8395–0.8440) at T1 (Table 2, Figure 2a); further ablations (T1-ns −0.0075 to −0.0086, T2 −~0.021) complete the curve.
Pre-specified and self-audited statistics
Non-inferiority margin δ=0.005 fixed before computation, paired DeLong under Holm with TOST equivalence, and a permutation audit of the DeLong implementation itself (6.0% type-I over 400 null draws, Supplementary Methods) — rare rigor for this literature.
One cohort subjected to every deployment audit simultaneously
Fairness repair benchmarked, not just proposed
Strong reproducibility posture
Weaknesses
Residual conditioning-on-care in the headline screening tier
Confirmatory inference confined to one test partition
Generalization from one data family to a whole literature
Asymmetric adaptation across model classes
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper attacks a specific, widely repeated number — AUROC ≈ 0.89 for machine-learning cardiovascular screening on the CDC's BRFSS survey — and asks how much of it survives controlled removal of leakage-prone predictors, whether the answer depends on the learner, and whether deployment-relevant properties (calibration, threshold-level fairness, group-conditional uncertainty, explanation faithfulness, cost, temporal transport) hold simultaneously on the same models. The central claim: the celebrated headroom is a property of the feature set, not the model class, and a glass-box EBM matches everything else once leakage is removed.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is target leakage in BRFSS-based heart disease prediction models?
Target leakage means training a model on features that are consequences of the outcome rather than antecedents of it. In BRFSS myocardial-infarction models, items like a prior angina diagnosis or having had a chest CT are recorded because of a diagnosis, so a model using them partly 'reads the answer back'. This paper shows those two items add 0.049–0.051 AUROC to the routinely reported ~0.89 across ten different model classes, without conferring real screening ability.
How accurate is cardiovascular screening on BRFSS once leakage features are removed?
After removing the two post-diagnostic items, all ten tested models (logistic regression through tabular foundation models) fall from ~0.89 AUROC into a tight band of 0.8395–0.8440 — a spread of only 0.0045. Removing prior stroke too costs a further ~0.008, and restricting to the 15 purely self-reportable items yields about 0.82 AUROC.
Is the explainable boosting machine (EBM) as accurate as XGBoost or TabPFN/TabICL on tabular health data?
In this study, statistically yes: at the cleaned feature tier, the EBM was non-inferior to all nine comparators within a pre-specified 0.005 AUROC margin, and it scored 88,413 respondents in 0.1 seconds versus 1,153.8 seconds of GPU time for TabICL — roughly four orders of magnitude faster — while its additive shape functions also enabled a fairness repair that matched conventional mitigation.
Why does the same screening model detect fewer heart attacks in women than in men?
At one threshold calibrated for ≥85% sensitivity overall, the model detected 75.4% of women's prior infarctions versus 89.0% of men's — a property of the data-plus-threshold system, not of any particular model (XGBoost showed the same 0.127–0.128 gap). The disparity runs through proxy variables (age, general health, smoking) interacting with sex; zeroing or attenuating the model's sex-related shape functions plus per-group intercept equalization shrank the gap to about 0.010, matching per-group thresholds and reweighing.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Superscripts are flattened in the PDF extraction: the abstract's 'roughly 104 times faster' corresponds to the Discussion's 'four orders of magnitude' (0.1 s vs 1,153.8 s, Figure 2b), i.e. 10⁴; likewise Table 1's '≥ 105-row training context' denotes ≥10⁵ (Methods: 'hundreds of thousands of rows').
- The TRIPOD+AI checklist is stated to accompany the submission as a separate file and is not part of the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.