Artificial intelligence surrogates for treatment effect estimation with before-and-after data
Frances Dean, Anna Neufeld, Joshua Barrios, Geoffrey H Tison, Ahmed Alaa
stat.ML · cs.LG
Quick verdict
Before-and-after AI-prediction surrogates identify ATT without controls; PPI debiasing fixes failed surrogacy. Careful theory, partly circular validation.
Summary
The paper proposes a framework for estimating the average treatment effect on the treated (ATT) when clinical outcomes are never observed in the treated population, using AI model predictions as surrogate outcomes from paired before-and-after measurements (e.g., ECGs). A time-aware predictor is trained on a treatment-free longitudinal cohort; the within-person difference of its predictions applied to pre- and post-treatment measurements identifies the ATT under six assumptions (consistency, positivity, Markov covariate sufficiency, counterfactual baseline sufficiency, comparability, statistical surrogacy). Because surrogacy is untestable, the authors adapt prediction-powered inference: a small labeled subset of treated patients plus an independent untreated sample corrects model error and surrogacy failure simultaneously, with covariate-shift extensions (Appendix C). Synthetic experiments (50 runs; linear and nonlinear autoregressive processes; surrogacy varied via rhoX1, rhoX2) show the naive estimator biased when surrogacy degrades (linear DGP, rhoX1=0.5, rhoX2=1: naive bias -0.529 vs PPI 0.027; PPI interval 0.631 vs difference-in-means 0.886), with roughly 1-10% of labels sufficing. A cardio-oncology analysis of 491,090 UCSF ECGs (anthracycline N=644, trastuzumab N=195; model AUROC 0.847 over 15 years) finds debiased surrogate estimates consistent with covariate-adjusted regression, with the naive surrogate faring better for anthracyclines. Main limitations: the naive estimator rests on an unverifiable constant-bias cancellation; the real-data benchmark regression itself assumes unconfoundedness; the treated cohort is described only as semi-synthetic.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under Assumptions 1-6 with a perfect outcome model, the within-person difference of AI predictions on pre- and post-treatment measurements identifies the ATT (and CATT) without any labeled treated outcomes (Propositions 1 and 4).
- A prediction-powered estimator combining a small labeled treated subset with an independent untreated labeled sample is asymptotically valid even when statistical surrogacy fails (Proposition 3), with covariate-shift extensions in Appendix C.
- In simulations, the naive surrogate estimator's bias tracks surrogacy degradation (naive bias -0.529 at rhoX1=0.5, rhoX2=1 vs PPI 0.027), and PPI intervals are narrower than difference-in-means even with 1% labels.
- In UCSF cardio-oncology data, debiased before-and-after AI surrogate estimates of anthracycline and trastuzumab heart-failure risk effects are consistent with covariate-adjusted regression, and the naive surrogate performs better for anthracyclines, consistent with its irreversible structural mechanism.
Strengths
Explicit, testability-labeled assumption architecture
Assumptions 1-6 are each tagged testable or untestable, tied to concrete diagnostics (overlap checks for positivity; two-measurement prediction-gain tests for Assumptions 3-4), and connected to Prentice surrogacy and Athey et al. 2019 in Appendix A (Table 4), which clarifies exactly what the framework trades (unconfoundedness) for what it adds (longitudinal surrogacy).
PPI debiasing is well integrated and empirically effective
Proposition 3 plus covariate-shift variants (Appendix C) correct both model error and surrogacy failure. Table 1 (linear DGP, rhoX1=0.5, rhoX2=1, 5% labels): naive bias -0.529 vs PPI 0.027; Table 2 shows PPI intervals narrower than difference-in-means even at 1% labels (1.407 vs 1.987); Table 3 shows 0.146 vs 1.998 for a great surrogate.
Confounding stress test demonstrates the core selling point
Large, clinically grounded real-data application
Simulation code released
Weaknesses
Naive estimator rests on fragile, unverifiable bias cancellation
Real-data validation is partly circular
Treated cohort construction is ambiguous
Sub-nominal coverage in several synthetic cells, undiscussed
No empirical head-to-head with positioned competitors
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.27180, stat.ML) asks when predictions from a pretrained AI outcome model can substitute for unobserved clinical endpoints in causal inference. The setting is distinctive: treated patients contribute paired high-dimensional measurements before and after treatment (ECGs in the motivating example) but never have the outcome observed; a treatment-free longitudinal cohort (Dtrain) supplies the measurement-to-outcome relationship. Deliverables: (i) Propositions 1 and 4 — under six labeled assumptions (consistency, positivity, Markov covariate sufficiency, counterfactual baseline sufficiency, comparability, statistical surrogacy) the within-person contrast of time-aware model predictions identifies ATT and CATT; (ii) Propositions 2-3 — inference either via a constant-bias cancellation argument (no labels) or via a prediction-powered (PPI) correction using a labeled subset of treated patients plus an independent untreated holdout; (iii) covariate-shift generalizations (Appendix C) and a two-stage CATT network (Appendix E). Claims are scoped honestly: surrogacy and comparability are repeatedly flagged as untestable.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can you estimate a treatment effect without a control group?
This paper argues yes, under conditions: each treated patient serves as their own control by comparing an AI model's outcome predictions made from measurements taken before and after treatment. Identification then requires six assumptions — notably that the AI-measured variable statistically surrogates the clinical endpoint and that the training and treated populations are comparable. Since surrogacy is untestable, the paper recommends always reserving a small subset of patients with true outcome labels to debias the estimate via prediction-powered inference.
What happens if the AI model is a poor surrogate?
The naive before-and-after estimator becomes biased. The paper's prediction-powered correction removes this bias in simulations using only about 1-10% labeled treated outcomes (e.g., naive bias -0.529 versus 0.027 after correction), at the cost of wider confidence intervals that shrink as the surrogate or the labels improve.
What data was used and is it available?
The real application uses 491,090 twelve-lead ECGs from UCSF (1980-2021) with EHR-derived heart-failure labels, studying anthracycline (N=644) and trastuzumab (N=195) cardiotoxicity in breast cancer patients. The ECG data cannot be shared due to data-use agreements, but the simulation code is public on GitHub (AlaaLab/before-and-after-surrogates-simulations), so the synthetic results are independently reproducible.
How does this differ from the surrogate index of Athey et al. (2019)?
Athey et al. estimate long-run treatment effects by regressing outcomes on short-term proxies, but require unconfoundedness between treated and control groups at the same time point. This paper instead uses within-person before/after contrasts on treated patients only, dropping the unconfoundedness assumption but adding longitudinal surrogacy and comparability assumptions; the two frameworks are presented as complementary.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 (positivity plot, cardio-oncology results, covariate-shift and label-quantity curves) are not analyzable from extracted text; cardio-oncology point estimates and CIs are unknown.
- Equation extraction is imperfect (e.g., estimator names rendered as [ATT); Proposition statements reconstructed from surrounding context.
- Several cited works are dated 2025-2026; their content could not be verified offline.
- The term semi-synthetic for the treated cohort is ambiguous in the source text itself, not only in this extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.