arXiv paper2026-09-02· machine learning — distribution-free uncertainty quantification· method paper

Occupancy-based Quantile Risk Control

Q: What is Occupancy-based Quantile Risk Control (OQRC)?

A post-hoc calibration method that guarantees quantile-based risk measures of an ML model (e.g., value-at-risk, CVaR, VaR-intervals) stay below a user-set threshold α with probability at least 1−δ, using only a finite calibration set. It partitions the loss range into bins defined by sorted calibration losses, treats how test losses fill these bins as an occupancy problem with an exactly known uniform distribution, and builds a confidence upper bound from worst-case (bin-maximum) losses.

Q: How does OQRC differ from conformal risk control (CRC)?

CRC controls only the expected loss. OQRC controls a broader class — quantile-based risk measures such as β-VaR, β-CVaR and VaR-intervals — with a finite-sample guarantee rather than a conservative or asymptotic one, and mathematically recovers CRC as the special case of a single test sample with expected loss (Theorem 5.1).

Q: Is OQRC still valid when losses have ties or are discrete?

Yes — validity holds even with ties: Appendix A.1 breaks ties using an augmented lexicographic order on (loss, input) pairs. What can degrade is tightness: the O_p(n^{-1/2}) convergence to the optimal bound assumes a continuous loss density bounded away from zero, and the paper shows (Appendix D) that coverage can sit slightly above 1−δ for discrete losses like the false-negative proportion.

Q: What data and models were used in the paper's experiments?

Three tasks: polyp segmentation on 798 images pooled from Kvasir, Hyper-Kvasir, CVC-ColonDB, CVC-ClinicDB and ETIS-Larib (PraNet model); multi-label classification on 2000 MS COCO samples (ResNet50, plus EfficientNet, TResNet and ConvNeXt for robustness checks); and text emotion recognition on 5427 GoEmotions examples (a publicly fine-tuned BERT). Calibration sets of 200 and test sets of 500, repeated 1000 times per configuration.

Zihao Shi, Huajun Xi, Bingyi Jing, Hongxin Wei

View original paper ↗arXiv:2609.03104

stat.ML · cs.LG

Quick verdict

OQRC claims tight, finite-sample quantile risk control via an occupancy UCB; elegant theory and strong gains, but MC slack and discreteness gaps remain.

Summary

The paper introduces Occupancy-based Quantile Risk Control (OQRC), a post-hoc calibration method that controls quantile-based risk measures (β-VaR, β-CVaR, VaR-Interval, expected loss) with finite-sample guarantees. Positioned against conformal risk control (expected loss only), quantile risk control (conservative bounds), and CDRC-L (asymptotic-only guarantees), OQRC partitions the loss range [L,U] into n+1 bins using ordered calibration losses, treats the counts of m test losses in these bins as an occupancy problem, and proves the occupancy vector is discrete-uniform over all compositions (Theorem 3.1). Raising each test loss to its bin maximum yields an upper risk functional whose (1−δ) quantile under this uniform law is a valid upper confidence bound (Theorems 3.2, 3.5); a stars-and-bars Monte-Carlo sampler with DKW control (Proposition 3.3, Theorem 3.4) makes the method tractable, and Theorem 3.6 shows the bound converges to the optimal at O_p(n^{-1/2}) when the loss density is bounded away from zero. Experiments on polyp segmentation (PraNet), MS COCO multi-label classification (ResNet50 plus three backbones), and GoEmotions (BERT), with FNP loss, n=200, m=500, and 1000 trials, show valid coverage (0.904–0.959) and RiskGap reductions up to 78.64% versus truncated Berk-Jones baselines, while CDRC-L fails coverage (~0.84). Limitations: i.i.d. setting only; tightness theory assumes continuous losses while experiments use discrete FNP, explaining coverage slightly above 1−δ (Appendix D); Monte-Carlo quantile error is not folded into the final guarantee.

Review complete

Assessment at a glance

machine learning — distribution-free uncertainty quantification

risk flags identified

● high 0

● medium 2

● low 2

Reproducibility readinessmedium

Code & data availablenot stated

Review confidencehigh

AI text signal0/20 passages flagged

Deep research viableCode-reproducibleNeeds GPU

5 strengths · 5 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

The occupancy vector of m test losses across n+1 calibration-induced bins is discrete-uniform over all compositions of m (Theorem 3.1), independent of the loss values.
OQRC's bin-maximum upper bound yields P(Rtest(λ*) ≤ α) ≥ 1−δ with finite-sample validity for any quantile-based risk measure (Theorems 3.2, 3.5).
The OQRC bound converges to the optimal (1−δ) bound at rate O_p(n^{-1/2}) when the loss density is bounded between c>0 and C<∞ (Theorem 3.6).
OQRC reduces RiskGap by up to 78.64% (MS COCO, VaR-Interval, vs Two-sided BJ) while keeping valid coverage, whereas CDRC-L fails coverage at n=200 (0.841–0.844 vs required 0.9).
The framework extends to unconditional risk control, recovering conformal risk control as the m=1, expected-loss special case (Theorem 5.1).

Strengths

Exact finite-sample occupancy foundation
Theorem 3.1 proves the test-loss occupancy vector is discrete-uniform over all (n+1)-part compositions of m — value-free, closed-form, with a clean exchangeability plus stars-and-bars counting proof; Appendix A.1 removes the NoTies restriction via a lexicographic order on (loss, input) pairs, matching the tied/discrete-loss setting actually used in experiments.
Validity without asymptotics, for a broad risk class
Theorems 3.2 and 3.5 give P(Rtest(λ*) ≤ α) ≥ 1−δ for any quantile-based risk measure (Definition 2.3: expected loss, β-VaR, β-CVaR, VaR-Interval), unlike CDRC-L [12] whose guarantee is asymptotic only — borne out empirically by CDRC-L coverage of 0.841–0.844 at n=200 in Table 1, below the required 0.9.
First tightness rate for quantile risk control
Unifying unconditional-control extension
Broad, well-instrumented evaluation

Weaknesses

Monte-Carlo plug-in gap in the deployed guarantee
Tightness theory contradicts the experimental loss
Narrow main-table evidence
Missing finite-sample testing baselines
Reproducibility not yet actionable

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

The paper targets post-hoc risk control for a black-box predictor whose tunable parameter λ trades utility against risk. Where conformal risk control (CRC; Angelopoulos et al. [6]) guarantees only the expected loss, and quantile risk control (QRC; Snell et al. [9]) covers quantile-based risk measures (QBRMs — expected loss, β-VaR, β-CVaR, [βmin, βmax]-VaR-Interval, Definition 2.3) with conservative order-statistic/Berk-Jones bounds, the authors propose OQRC: tight QBRM control with finite-sample validity. Central claims: (i) the occupancy vector of m test losses across n+1 calibration-induced bins is discrete-uniform over all compositions (Theorem 3.1); (ii) the bin-maximum bound R+(λ) yields P(Rtest(λ*) ≤ α) ≥ 1−δ (Theorems 3.2, 3.5); (iii) R+ converges to the optimal (1−δ) bound at rate O_p(n^{-1/2}) when the loss density is bounded away from zero (Theorem 3.6); (iv) empirically, RiskGap drops up to 78.64% versus truncated Berk-Jones while coverage stays valid, and the L-statistics competitor CDRC-L fails coverage at n=200. A fifth contribution extends the machinery to unconditional control, recovering CRC as the m=1, expected-loss special case (Theorem 5.1, Section 5).

Methodology Assessment

The construction is clean and the extracted proofs are coherent. Partitioning [L,U] by calibration order statistics (Eq. 3) turns the problem into classical occupancy; the uniformity theorem is proved by exchangeability of the pooled sample plus stars-and-bars counting, with the NoTies restriction removed in Appendix A.1 via a lexicographic order on (loss, input) pairs — a practically useful relaxation matching the discrete FNP loss. Validity (Theorem 3.2) follows from rank uniformity and the inclusion {Φ+(N) ≤ Φ+,(k)} ⊆ {Rtest ≤ Φ+,(k)}; λ selection (Theorem 3.5) re

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 5 strengths & 5 weaknesses, analyzed in detail
4 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is Occupancy-based Quantile Risk Control (OQRC)?

A post-hoc calibration method that guarantees quantile-based risk measures of an ML model (e.g., value-at-risk, CVaR, VaR-intervals) stay below a user-set threshold α with probability at least 1−δ, using only a finite calibration set. It partitions the loss range into bins defined by sorted calibration losses, treats how test losses fill these bins as an occupancy problem with an exactly known uniform distribution, and builds a confidence upper bound from worst-case (bin-maximum) losses.

How does OQRC differ from conformal risk control (CRC)?

CRC controls only the expected loss. OQRC controls a broader class — quantile-based risk measures such as β-VaR, β-CVaR and VaR-intervals — with a finite-sample guarantee rather than a conservative or asymptotic one, and mathematically recovers CRC as the special case of a single test sample with expected loss (Theorem 5.1).

Is OQRC still valid when losses have ties or are discrete?

Yes — validity holds even with ties: Appendix A.1 breaks ties using an augmented lexicographic order on (loss, input) pairs. What can degrade is tightness: the O_p(n^{-1/2}) convergence to the optimal bound assumes a continuous loss density bounded away from zero, and the paper shows (Appendix D) that coverage can sit slightly above 1−δ for discrete losses like the false-negative proportion.

What data and models were used in the paper's experiments?

Three tasks: polyp segmentation on 798 images pooled from Kvasir, Hyper-Kvasir, CVC-ColonDB, CVC-ClinicDB and ETIS-Larib (PraNet model); multi-label classification on 2000 MS COCO samples (ResNet50, plus EfficientNet, TResNet and ConvNeXt for robustness checks); and text emotion recognition on 5427 GoEmotions examples (a publicly fine-tuned BERT). Calibration sets of 200 and test sets of 500, repeated 1000 times per configuration.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Figures 3 and 5–17 are image-only in the extracted text; appendix sweep and synthetic-coverage results could only be assessed qualitatively, not numerically.
No code URL appears anywhere in the text; codeAvailable=false reflects the checklist's promise to release ('We will release'), not an existing release.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper