Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores
Hyeonsu Lee, Juyeon Kim, Erkhembayar Jadamba, Seungjin Choi, Hyunjin Shin
cs.LG
Quick verdict
Tilting-identity fix aligns Bayesian conformal scores with label shift; derivation and experiments are clean, but finite-sample validity stays approximate.
Summary
The paper targets conformal prediction under label shift, where the label marginal changes while p(h|y) is stable. Existing importance-weighted conformal methods pair reweighting with residual scores, giving uniform-width intervals, while Bayesian conformal methods use adaptive predictive-density scores evaluated under the source distribution, which is misaligned with the target. The authors derive a posterior predictive tilting identity (Proposition 2.1) showing the target predictive equals an importance-weighted transform of the source predictive, yielding a decomposition into source score, shift correction, and normalization. Instantiated with Bayesian Ridge Regression and a log-linear density ratio, the tilted predictive stays Gaussian with a shifted mean μ*(h) = μ(h) + β̂₁σ²(h), giving closed-form adaptive intervals at no extra cost. Experiments on AqSolDB (9,982 compounds; frozen 171.16M-parameter molecular BART encoder; BRR head) resample test sets via exponential tilts β ∈ {0, −0.1, …, −0.5} with 1,000 seeds per configuration. At β = −0.5, LSA intervals average 4.98 versus 5.74 (source Bayesian score) and 5.82 (residual) — a 13–14% reduction — with comparable coverage (.8842 vs .8795/.8797, all below nominal 0.9). Lipophilicity replicates the trend, as does a Gaussian-mean weight-estimator ablation. Main limitations: no finite-sample validity claim under shift (pseudo-label density ratios), only negative one-parameter tilts evaluated, and applicability confined to a linear-Gaussian head on frozen features.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under label shift, the target posterior predictive equals an importance-weighted transform of the source predictive (Proposition 2.1), which directly motivates a corrected Bayesian nonconformity score.
- With a Gaussian predictive and log-linear density ratio, the LSA correction reduces to a predictive-mean shift μ*(h)=μ(h)+β̂₁σ²(h) and admits closed-form adaptive prediction intervals at no computational overhead.
- On AqSolDB under exponential-tilt shifts down to β=−0.5 (1,000 seeds), the LSA score yields 13–14% shorter intervals than source-score and residual baselines at comparable target coverage, with the trend replicated on Lipophilicity.
- Residual coverage loss under stronger shift affects all methods and stems from pseudo-label-based density-ratio estimation, not from the target-aligned score itself.
Strengths
Clean theoretical core with full proof
Proposition 2.1 (posterior predictive tilting) is derived end-to-end in Appendix B under explicit Assumptions 2.1–2.3; eq. (4) decomposes the score into source score, shift correction, and normalization, and Appendix F gives exact score inversion plus the clipped implementation actually used.
Closed-form practical instantiation
With BRR and a log-linear estimated ratio, the tilted predictive remains Gaussian with shifted mean μ*(h)=μ(h)+β̂₁σ²(h) and unchanged variance (eq. 13), so eq. (16) yields adaptive intervals at zero extra cost — verified analytically in Appendices D–E.
Genuinely controlled comparison
Robustness ablations
Honest scope framing
Weaknesses
No oracle-weight control experiment
Pseudo-label circularity
Narrow shift regime
Coverage claims rest on overlapping distributions
Limited model scope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This ICML 2026 workshop paper (EIML@ICML) addresses split conformal prediction under label shift, where the label marginal changes but p(h|y) is stable. The claimed gap: importance-weighted conformal methods (Tibshirani et al. 2019; Podkopaev & Ramdas 2021) restore calibration using residual scores that ignore predictive uncertainty (uniform widths), while Bayesian conformal scores (Fong & Holmes 2021) are adaptive but evaluated under the source predictive, misaligned with the target. The contribution is the Label-Shift-Adjusted (LSA) score, derived from a posterior predictive tilting identity (Proposition 2.1, proof in Appendix B): under conditional-label-shift, absolute-continuity, and source-representativeness assumptions, p_t(y|h,D_train) = p_s(y|h,D_train)·r(y)/Z(h), yielding the decomposition of eq. (4) — source score plus a −log r(y) correction plus normalization. The authors are explicit that no exact finite-sample validity under shift is claimed (ratios come from pseudo-labels); the contribution is efficiency — shorter intervals at comparable empirical coverage.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the LSA score in conformal prediction?
The Label-Shift-Adjusted Bayesian Score is a nonconformity score that corrects the standard Bayesian score (negative log predictive density) for label shift. Using an identity showing the target predictive distribution is an importance-weighted version of the source predictive, it adds a −log density-ratio correction plus a normalization term, so conformity is measured against the target distribution rather than the source one.
Does the LSA method guarantee coverage under label shift?
No. The paper explicitly does not claim finite-sample validity under shift because the density ratio is estimated from pseudo-labels. Empirically all methods — including LSA — fall below the nominal 0.9 coverage as shift grows (e.g., about 0.884 for LSA at β=−0.5); the demonstrated benefit is shorter intervals at roughly equal coverage.
How much shorter are the prediction intervals with the LSA score?
On the AqSolDB solubility benchmark with an exponential-tilt label shift of β=−0.5, average interval length drops to about 4.98 versus 5.74 (source Bayesian score) and 5.82 (residual score) — roughly 13–14% shorter — averaged over 1,000 random seeds with comparable coverage. On the Lipophilicity benchmark the reduction is smaller, about 4.6–5.3%.
What data and models were used in this paper?
A Bayesian Ridge Regression head was fitted on frozen 768-dimensional representations from a custom pretrained molecular BART-style language model (171 million parameters), evaluated on AqSolDB aqueous solubility (9,982 compounds) and a Lipophilicity benchmark, with test sets resampled under exponential-tilt label shifts β from 0 to −0.5. No code release is mentioned in the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 are represented only by their captions in the extraction; the plotted curves could not be inspected, though the underlying per-β numbers are available in Tables 1–3.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.