Memorisation bias in medical AI
Moritz A. Knolle, Martin J. Menten, Laurin Lux, Mélanie Roschewitz, Emma A. M. Stanley, Georgios Kaissis, Daniel Rueckert, Ben Glocker
cs.LG · cs.CY
Quick verdict
Claims training on patients' past records biases AI predictions on their future data; rigorous large-scale evidence, but clinical harm is modest.
Summary
The authors ask whether medical AI models trained on patients' historical de-identified records change predictions on those same patients' unseen future records, a phenomenon they name memorisation bias. Using MIMIC-ECG, MIMIC-CXR, MIMIC-IV-ED and the Harvard-Emory ECG database, they train M=200 models per dataset on balanced random 50% patient subsets, partition models per patient by inclusion, and compare prediction distributions with energy-based tests (BH-FDR). Roughly 0.6-4.7% of future records show significant shifts, with effect sizes sometimes exceeding 70 percentage points; randomly partitioned controls flag zero records. Significant shifts persist for decades (HEEDB, >25 years). In simulated deployment with Youden thresholds, sensitivity drops for de novo conditions whereas sensitivity and specificity both inflate when health state is unchanged. Among tabular models, random forests memorise most. Patient-level differential privacy nearly eliminates the effect while record-level DP does not, even at epsilon=1. Limitations: the label-informed adaptive temporal split enriches future data for de novo cases; the harm analysis conditions on flagged records; absolute effect sizes and missed-diagnosis counts are modest; the patient-level DP result partially conflates the privacy unit with a 3-4x training-data reduction; DP experiments use a downsized ViT-T model.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Including a patient's historical records in training systematically shifts model predictions on their unseen future records toward the historical health state, across ECG, chest X-ray and EHR modalities (0.6-4.7% of future records flagged; effect sizes up to over 70 percentage points).
- Memorisation bias persists over long horizons, with significant prediction changes on records acquired more than 25 years after the patient's last historical training record (HEEDB).
- In simulated prospective deployment, diagnostic sensitivity is significantly reduced for de novo conditions, whereas sensitivity and specificity are both significantly inflated when a returning patient's health state is unchanged.
- Random forests are far more prone to memorisation bias than logistic regression or tabular ResNets on MIMIC-IV-ED (2.25% vs 0.04% and 0.12% of future records flagged).
- Patient-level differential privacy essentially eliminates memorisation bias on future records, while record-level DP leaves substantial residual bias even at the strongest tested budget (epsilon=1).
Strengths
Novel, clinically consequential framing
First systematic study of longitudinal memorisation effects on real-world patient data (Main; Discussion), reframing memorisation from a privacy-attack concern into a deployment-safety risk for returning data contributors, with concrete regulatory context (EU AI Act Art. 10(4), national screening programmes).
Strong experimental design with negative controls
5,200 models trained on balanced 50% patient subsets (Carlini-style design guaranteeing 100 vs 100 models per patient); random-partitioning baselines flagged zero records on both historical and future data across all datasets (Fig. 3, Extended Data Fig. 1b), isolating patient inclusion as the only variable.
Statistical rigor throughout
Breadth across modalities, architectures and timescales
Candid limitations and actionable mitigation analysis
Weaknesses
Post-selection inference in the harm analysis
Label-informed adaptive temporal split limits generalisability
Patient-level DP comparison is confounded
Modest absolute effects versus abstract-level framing
Mechanistic claim untested; single-institution data
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is memorisation bias in medical AI?
It is the phenomenon where an AI model trained on a patient's past de-identified medical records makes systematically different predictions on that same patient's future records, pulling predictions toward the health state seen during training. In this paper's four large datasets, roughly 0.6-4.7% of future records were significantly affected, occasionally with prediction changes exceeding 70 percentage points.
Does anonymising patient data prevent memorisation bias?
No. Anonymisation removes the ability to identify whose records are in a model's training set, so hospitals cannot easily exclude returning patients from AI-assisted interpretation of their own future data. The authors argue that de-identification protects privacy but simultaneously obscures who is exposed to memorisation-induced diagnostic errors.
Can differential privacy fix memorisation bias in medical AI models?
It depends on the privacy unit. In the paper's experiments, record-level differential privacy reduced but did not eliminate memorisation bias even at a strong budget (epsilon=1), while patient-level differential privacy, which jointly protects all of a patient's records, almost completely removed it on future records. However, the patient-level implementation used here discarded all but one record per patient, so part of its utility cost reflects data loss rather than the privacy mechanism itself.
Should patients worry that AI models memorised their medical records?
The detected effects are real and directionally unfavourable: higher missed-diagnosis risk when a patient develops a new condition, and artificially inflated accuracy when their health state is unchanged. However, the absolute number of affected predictions in these simulations was modest, and true clinical impact requires prospective study. The concern is expected to grow as models and training populations scale up.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are text-mangled in the extraction; exact point estimates (e.g., Fig. 4 sensitivity differences, ecCDF shapes) could not be verified, only ranges inferred from axis labels and quoted counts.
- The metadata abstract is truncated mid-sentence, but the complete abstract appears in the extracted full text.
- Supplementary Material Table 1 is included, but referenced Extended Data panels could only be assessed from captions, not visually.
- External claims (arXiv ID 2609.17223, GitHub repository contents, cited 2026 references) could not be cross-checked offline.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.