Learning Risk Scores Robust to Unobserved Confounders
Ryan Edmonds, Yingxiao Ye, Sina Aghaei, Andrés Gómez, Çağıl Koçyiğit, Phebe Vayanos
cs.LG
Quick verdict
Risk-score learning robust to unobserved confounding via MSM+Wasserstein uncertainty set, cone reformulation; sound but only semi-synthetically validated.
Summary
The paper addresses learning risk scores — linear models estimating the probability of an adverse outcome under no treatment — from observational data whose historical treatment assignments were shaped by unobserved confounders, which invalidates standard inverse propensity weighting (IPW). The authors treat the unknown propensity weights as uncertain parameters in a set combining Tan's marginal sensitivity model odds-ratio bound (Γ), a Wasserstein-distance consistency constraint aligning the reweighted untreated population with the observed covariate distribution, a normalization constraint, and a one-sided 'historical rationality' constraint related to the enhanced MSM. The resulting max-min log-likelihood problem is reformulated exactly as an exponential cone program solvable with off-the-shelf solvers (MOSEK). On 15 UCI datasets converted to semi-synthetic confounded data via an outcome-dependent treatment policy with strength γ, the method reportedly improves calibration error by up to 29.2% relative to doubly robust estimation and up to 11.1% relative to an MSM-only robust baseline (the Kallus–Zhou uncertainty set), with best performance when Γ lies between e^γ and e^2γ. Limitations: evaluation is exclusively semi-synthetic (the motivating homelessness setting is untested); the rationality constraint mirrors the synthetic treatment generator; Γ and ε selection requires oracle-like knowledge of confounding strength; and only mean metrics over 15 replications are reported, without dispersion or significance testing.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under unobserved confounding, propensity weights can be treated as uncertain parameters constrained by MSM odds-ratio bounds (C3), Wasserstein consistency with the observable covariate marginal (C2), scale normalization (C1), and a one-sided historical-rationality constraint (C4).
- The resulting max-min log-likelihood risk score problem admits an exact sample-based reformulation as an exponential cone program (Proposition 2) solvable with off-the-shelf solvers such as MOSEK.
- Adding the Wasserstein consistency and rationality constraints to the MSM-only set matters because the latter admits reweightings incompatible with the observed covariate distribution, as shown by a counterexample in Section 2.
- On 15 semi-synthetic UCI datasets, with Γ between e^γ and e^2γ, the method reduces expected calibration error by up to 29.2% relative to doubly robust estimation and up to 11.1% relative to the MSM-only robust baseline.
Strengths
Well-argued uncertainty-set design with a falsifying counterexample
Section 2's example (P(X=±1)=0.5, Z|X=−1=0.25, Z|X=1=0.75) shows a Z satisfying C1/C3 that distorts the reweighted covariate distribution to 0.75/0.25, demonstrating that the MSM-only set of Kallus and Zhou (2021) admits distributions incompatible with observable data; using 1-Wasserstein distance for C2 is justified over φ-divergences by finite-sample support mismatch.
Exact tractable reformulation
Proposition 2, derived in EC.2.3 by casting the Wasserstein constraint as an optimal transport problem and applying strong duality, yields an exponential cone program solvable with MOSEK; Proposition 1 and Lemma 1 proofs in EC.2 verify the population-level consistency of C1–C2 at the true propensity.
Honest sensitivity reporting and sanity checks
Clear positioning against the sensitivity-analysis literature
Weaknesses
Semi-synthetic-only evaluation
Structural alignment between the rationality constraint and the synthetic generator
Oracle-like hyperparameter knowledge
No dispersion or significance reporting
Scalability and runtime untested
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets risk-score learning from observational data (xi, ti, yi) where the historical allocation policy depended on unrecorded information, breaking unconfoundedness and invalidating inverse propensity weighting (IPW). The target is the conditional distribution of the no-treatment potential outcome Y0 given X, modeled as logistic in a linear score β⊤x (Section 2). Claimed contributions: (i) an uncertainty set over propensity weights combining Tan's marginal sensitivity model (MSM) odds-ratio bound (C3), marginal-scale normalization (C1), a Wasserstein consistency constraint against the observable covariate marginal (C2), and a one-sided 'historical rationality' constraint (C4) related to the eMSM outcome bound of Zhang et al. (2025); (ii) an exact sample-based exponential-cone reformulation (Proposition 2) solvable with MOSEK; (iii) experiments on 15 semi-synthetic UCI datasets showing calibration improvements of up to 29.2% vs doubly robust (DR) and 11.1% vs an MSM-only robust baseline. All three claims are supported at the level of the presented evidence, but the evidence is exclusively semi-synthetic.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does this paper solve?
It learns risk scores — simple linear models estimating a person's probability of an adverse outcome if left untreated — from historical records where past treatment decisions depended on information that was never recorded. That unrecorded information (unobserved confounding) biases standard correction methods like inverse propensity weighting, so the authors instead optimize the score against the worst plausible assignment of treatment propensities within a bounded uncertainty set.
How should the robustness parameter Γ be chosen?
Γ bounds how much the true odds of receiving no treatment may differ from the odds predicted from recorded characteristics; Γ=1 means no hidden bias. In the paper's semi-synthetic experiments the best results occur when Γ is between e^γ and e^2γ, where γ measures the true confounding strength — which is unknown in practice. The authors suggest domain experts calibrate Γ (following the sensitivity-analysis literature), but no data-driven selection procedure is provided or tested.
What data and code are used in the experiments?
The experiments use 15 UCI Machine Learning Repository datasets (e.g., Abalone, Adult, Covertype, Spambase), converted to semi-synthetic confounded data by simulating treatments from a logistic policy that depends on the true outcome. No public code release is mentioned; the method is implemented as an exponential cone program solved with the MOSEK solver on 1500 training and 500 test points per dataset.
How much better is this method than existing approaches?
In Table 1, when Γ is set near the true confounding strength, the method's expected calibration error is up to 29.2% lower than doubly robust estimation and up to 11.1% lower than an MSM-only robust baseline across the tested confounding levels. These are means over 15 replications on semi-synthetic data with no confidence intervals reported, and the motivating real-world setting (homelessness services) was not evaluated.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Mathematical notation lost sub/superscripts in extraction (e.g., Proposition 2's dual variables appear as duplicated symbols θi/θi), so the exact cone formulation could not be verified line-by-line.
- No figures are present in the extraction; Table 1 values were used as printed (3-decimal rounding), so the derived 29.2% and 11.1% figures are recomputed from rounded means.
- Whether Table 1 averages across the 15 datasets or reports one aggregate is not stated in the extracted text.
- Appendix EC.3.3's Γ–γ approximation argument could only be checked at the level of the displayed inequalities (Jensen bounds e^{±2γ}).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.