AUC Maximization from Biased Positive-unlabeled Data with Confidence
Atsutoshi Kumagai, Tomoharu Iwata, Hiroshi Takahashi, Taishi Nishiyama, Kazuki Adachi, Yasuhiro Fujiwara
cs.LG · cs.AI · stat.ML
Quick verdict
Proven AUC maximization from biased PU data reweights labeled positives by confidence ratio; strong results on 8 datasets, but relies on ordered confidence.
Summary
This paper studies AUC maximization — the standard ranking metric for imbalanced binary classification — from positive-unlabeled (PU) data in the realistic selected-at-random (SAR) setting, where the probability that a positive instance gets labeled depends on its features. The key ingredient is confidence — the probability of being positive — attached to the few labeled positives. The authors derive an AUC risk estimator that reweights each labeled positive by the ratio of its confidence to its labeling probability; only the label ratio and the confidences enter, so the positive class prior and labeling rate are not required. Two theoretical results are given: the Bayes-optimal AUC ranking is preserved even if the given confidence is any strictly increasing transformation of the true posterior, and a generalization bound holds for finite function classes under bounded confidence error and a lower-bounded labeling probability. On eight real-world datasets — six with simulated selection bias and two with human-annotator confidence (Cifar10-H, Fmnist-H) — the method is competitive with nine baselines, and ablations show robustness to monotone confidence distortions and moderate noise. Main limitations: confidence must be correct at least in ordering, every positive region is assumed to have positive labeling probability, the main experiments rely on an extra 10,000 labeled instances to synthesize confidence, and the theory covers finite function classes rather than the neural networks used experimentally.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The derived AUC risk estimator (Eq. 17) reweights each labeled positive by confidence divided by labeling probability, and enables AUC maximization under the SAR setting without knowing the class prior π or labeling rate c.
- Minimizing the rewritten risk still yields the Bayes-optimal AUC ranking when the observed confidence is any strictly increasing transformation of the true posterior (Proposition 4.1).
- The empirical objective with estimated labeling probability û approximates the true AUC risk within ε/((1−π)δ), giving an excess-risk guarantee (Proposition B.2, Theorem B.4).
- Across eight real-world datasets, including two with real human-annotator confidence, the method performs best or statistically comparably to the best method against nine baselines (Tables 1, 2, 11).
- No prior work performs AUC maximization from PU data under the SAR setting, and positive-confidence has not previously been used for PU learning under that setting (Sections 1-2).
Strengths
Novel problem positioning with a practical information source
Section 1/2 argue no prior work does AUC maximization from PU data under the SAR setting, and the required positive-confidence (r(x)=p(y=1|x)) is plausibly a by-product of multi-annotator labeling, BI-RADS scores, or alert severities — a credible motivation distinct from prior SAR-PU methods that assume restricted propensity forms.
Clean and self-contained estimator derivation
Section 4.2 decomposes the AUC risk (Eq. 10-12), rewrites p_p via the propensity relation pl=e/c·pp (Eq. 13, proven in C.2), and plugs in e(x)=p(o=1|x)/p(y=1|x) (Eq. 14, C.3); the final objective (Eq. 17) needs only û(x) and α̂, letting training proceed without π or c. Lemma B.1 carefully handles the u(x)=0 support condition; extensions cover non-symmetric surrogates (D.1) and the two-sample scenario (D.2).
Strong theoretical package beyond the estimator
Experiments target the theory's failure modes
Honest accounting of computation and degenerate cases
Weaknesses
Positivity assumption is strong and undiscussed for the motivating scenarios
Confidence quality is the load-bearing assumption, validated mostly in-distribution
Comparison fairness: baselines cannot access the auxiliary labeled set
Human-confidence evidence is partly degenerate
Theory-experiment gap and scattered numeric inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: AUC Maximization from Biased Positive-unlabeled Data with Confidence
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does 'AUC maximization from biased positive-unlabeled data with confidence' solve?
It learns imbalanced binary classifiers measured by AUC when you only have positive and unlabeled data, the labeled positives are a biased subset (the probability of being labeled depends on the instance's features, the SAR setting), and each labeled positive additionally carries a confidence value — the probability that it is truly positive. Negative labels are never available, which is common in security, medical, and fraud applications.
Why does the method need positive-confidence, and what if it is miscalibrated?
The confidence lets the method correct the sampling bias: each labeled positive is reweighted by its confidence divided by its estimated labeling probability. The paper proves the Bayes-optimal AUC ranking is preserved even if the given confidence is any strictly increasing transformation of the true posterior probability — so systematic over- or under-confidence is fine as long as the ordering is correct; random noise degrades performance only moderately (Gaussian noise with std up to 0.15 was tested).
How does this method differ from existing PU learning methods for biased positives (SAR)?
Prior SAR-PU methods (e.g., PUSB, PG, LBE) target ordinary classification and need restrictive assumptions such as data separation, specific propensity functional forms, or order invariance; none target AUC. This method replaces those assumptions with the availability of confidence on labeled positives and is designed to maximize AUC directly, while also not requiring the class prior or labeling rate.
What were the experimental results and are the datasets public?
On eight real, publicly available datasets (Mnist, Fmnist, Svhn, Cifar10, Diabetes, Blood plus Cifar10-H and Fmnist-H with human annotator confidence), the proposed method was best or statistically comparable to the best among nine baselines; for example an average test AUC of 0.8979 on Mnist and 0.8643 on Cifar10 (Table 1). Selection bias was injected synthetically for most datasets, and the paper does not mention releasing code.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided extraction ends immediately after the last data row of Table 11 (full appendix results); any content beyond that point was not available for review, though all main-text sections, proofs, and appendices A-F appear complete.
- Exact character counts of the non-English summaries were not programmatically verified; they were drafted to match the English summary's scope, numbers, and limitation items.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.