Attributing Cohen's d: Training Data Attribution for Disease-Related Effects in Normative Age Biomarkers
cs.LG · q-bio.QM
Quick verdict
Closed-form Cohen's d attribution to training samples; rigorous paired controls show robust gains, but missing baselines temper the contamination story.
Summary
The paper reframes training data attribution for normative age biomarkers: instead of attributing prediction loss, the authors derive a closed-form influence functional that ranks training samples by their effect on the downstream disease-related effect size (Cohen's d of the age gap between held-out cases and matched controls). For ridge regression the functional is exact up to the usual first-order approximation and is validated against leave-one-out retraining, with ranking overlaps of 98.9–99.8% and realized versus predicted changes of at most 0.009. On UK Biobank, across four diseases (multiple sclerosis, cerebrovascular disease, type 2 diabetes, chronic kidney disease) and two modalities (1,440 T1 imaging phenotypes; 251 NMR metabolites), removing the top 10% most influential training samples raises held-out Cohen's d in every one of ten seeds, while count-matched random removal leaves d flat even at 50% removal. Gains are largest for the metabolomic conditions: type 2 diabetes rises from 0.601 to 1.439 at 10% removal. Flagged subjects show subclinical cardiometabolic burden (for diabetes, HbA1c, d≈0.16), though chronic kidney disease flags ethnicity instead, raising fairness concerns. Main limitations: ridge-only, single biobank, per-condition reference cohorts unlike common practice, no comparison to loss-based attribution or simple phenotype-threshold exclusion, and an untested causal link between the gains and the contamination mechanism.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Attribution can target the downstream disease-related effect size (Cohen's d of the age gap) instead of training loss, via a closed-form functional for ridge regression validated against exact leave-one-out retraining (98.9-99.8% ranking overlap, realized change at most 0.009).
- Removing the 10% most influential training samples raises held-out Cohen's d in every one of ten seeds across four diseases and two modalities, while count-matched random removal leaves it flat even at 50% removal.
- The metabolomic-age effect for type 2 diabetes more than doubles at 10% removal (0.601 to 1.439), and the brain-age effect for multiple sclerosis rises by roughly a third (0.671 to 0.871).
- Flagged training subjects carry subclinical cardiometabolic burden missed by diagnosis-based exclusion (T2D: HbA1c, d=+0.162), consistent with the reference-cohort contamination hypothesis.
- For chronic kidney disease the strongest flagged separation is ethnicity (d=-0.077), an unresolved fairness concern.
Strengths
Genuinely novel attribution target
Instead of the training loss, the paper attributes the downstream evaluation statistic itself. Section 2 derives the closed-form functional carefully: F acts on the bias-corrected gap g, not raw predictions, so the gradient differs from standard influence functions (Eq. 1-2), and the fixed correction constants surface any induced error as a predicted-vs-realized gap that is then measured.
Exact ground-truth validation of the scores
Because the model is ridge, leave-one-out retraining is tractable for every training sample: rankings overlap 98.9-99.8%, and retraining under each ranking changes evaluation-half Cohen's d by a mean of 0.001 and a maximum of 0.009 across all condition/seed/removal combinations (Section 2, Validation).
Disciplined experimental controls
Phenotypic validation with a positive control and honest nulls
Unusual reproducibility support for a UKBB paper
Weaknesses
Missing decision-relevant baselines
Contamination mechanism not causally isolated
Effect magnitudes at high cuts strain plausibility
Design diverges from common practice
Informal statistical inference
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a known weakness of normative age biomarkers (brain age, metabolomic age): reference-cohort contamination — nominally healthy training cohorts containing undiagnosed or pre-diagnostic disease. The claimed contributions (Section 1) are: (i) Cohen's d influence functions, a closed-form attribution of the downstream disease-related effect size rather than the training loss; (ii) validation across four UK Biobank conditions (multiple sclerosis, cerebrovascular disease, type 2 diabetes, chronic kidney disease) and two modalities (1,440 T1 imaging phenotypes; 251 NMR metabolites); (iii) phenotypic characterisation of flagged samples. The core empirical claims are specific and falsifiable: removing the top 10% of training samples by influence raises held-out Cohen's d in every seed (ten per condition), count-matched random removal leaves d flat even at 50%, and type 2 diabetes more than doubles its effect (0.601 to 1.439 at 10% removal, Table 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a Cohen's d influence function in this paper?
It is a closed-form score that ranks training samples by their estimated effect on the disease-related effect size of a normative age model — the Cohen's d of the age gap between held-out disease cases and matched controls — rather than on prediction loss. For ridge regression it is computed analytically, and the authors validate it against exact leave-one-out retraining, finding ranking overlaps of 98.9-99.8%.
Does removing influential training samples actually improve disease detection?
In the paper's UK Biobank experiments, yes: removing the top 10% most influential training subjects raised the held-out case-control effect size in all ten random seeds across four diseases and two biomarker modalities, while removing the same number of subjects at random changed almost nothing even at 50% removal. The largest gain was for type 2 diabetes on metabolomic age, where the effect rose from 0.601 to 1.439 at 10% removal.
What data and code are needed to reproduce this study?
UK Biobank data under an approved application (the authors used project 33073), plus the authors' MIT-licensed Python packages pyinfluence (influence functions for scikit-learn) and cohortmatch (statistical matching), both on Zenodo. Appendix C documents cohort sizes, matching, splits, seeds, and compute: CPU-only, roughly 12.5 core-hours for the final experiments.
What are the main criticisms or concerns about this method?
It is validated only for ridge regression on a single biobank; it lacks comparisons against loss-based attribution methods and simple phenotype-threshold exclusions; its per-disease reference cohorts differ from the common shared-cohort practice, so practical gains may shrink; inference relies on sign consistency across ten seeds without formal tests; and for chronic kidney disease the flagged samples differ by ethnicity, an unresolved fairness concern.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Values shown inside figures (Figures 1-6) are only recoverable through captions and extracted axis labels; plot-level detail beyond the numbers reported in captions and tables could not be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.