A Ranking Approach for Measuring Calibration
Anirban Chatterjee, Rina Foygel Barber
stat.ME · cs.LG · stat.ML
Quick verdict
Proposes rankECE, a tuning-free rank-based miscalibration measure; strong theory and honest limits, but approximation is benchmarked only against binned ECE.
Summary
The paper introduces rankECE, a rank-based measure of miscalibration for binary classifiers, built by averaging residual products (Y−Z)(Y−Z') over consecutive order statistics of the predicted probabilities Z=f(X), motivated by the identity ℓ2-ECE = E[(Y−Z)(Y′−Z)]. The theory is the paper's core: the sample estimator concentrates around its mean with no assumptions on f (Prop 2.1, McDiarmid); 0 ≤ rankECE ≤ ℓ2-ECE (Prop 3.1); consistency for fixed f (Prop 3.2); a gap of (∥r∥TV+1)/n governed by the total variation of the residual function (Prop 3.3); and rankECE = 0 if and only if the model is perfectly calibrated (Thm 3.1). Compared with binned ECE, rankECE dominates up to a 4K/n correction (Thm 4.1), while binned estimators carry bias up to K/n (Lemma 4.1). Finite-sample and asymptotic calibration tests are derived, and ties are handled via a randomized distributional transform (Appendix F). Experiments on synthetic functions and four Hugging Face sentiment models (Amazon/Yelp Polarity) show rankECE tracks ℓ2-ECE more closely than binned variants for K ∈ {10, √n, n^(1/3), n/20}; rankECE tests control type-I error and run orders of magnitude faster than SKCE tests. Limitations: rankECE is a one-sided lower bound whose accuracy needs bounded variation of an unknown residual function (counterexample in Appendix C), the measure is n-dependent, and approximation comparisons are restricted to binned ECE.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 1
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- rankECE, defined via residual products of consecutive order statistics of f(X), is a tuning-free calibration measure whose sample estimator concentrates without any assumptions on f (Prop 2.1).
- rankECE = 0 if and only if the predictor is perfectly calibrated (Thm 3.1), and it approximates ℓ2-ECE within (∥r∥TV+1)/n when the residual has bounded variation (Prop 3.3).
- Up to a 4K/n correction, rankECE dominates the binned approximation ℓ2-binECE (Thm 4.1), whose estimation bias reaches K/n when the number of bins is not o(n) (Lemma 4.1).
- rankECE-based hypothesis tests control type-I error and perform competitively with SKCE-based tests at a small fraction of the computational cost (§5.2, Table A.6).
Strengths
Assumption-free estimation guarantee
Prop 2.1 (via McDiarmid, bounded-difference constant 2.25/n in Lemma D.1) shows rankECE_n concentrates around its mean for any predictor f — the testability property of Rossellini et al. (2025) that ℓ2-ECE cannot have given the hardness result of Remark 3.2.
Complete theoretical chain around a one-line estimator
Lower bound 0 ≤ rankECE ≤ ℓ2-ECE (Prop 3.1), consistency for fixed f (Prop 3.2), TV-based gap (∥r∥TV+1)/n (Prop 3.3), iff calibration characterization (Thm 3.1, n ≥ 4), dominance over binned ECE up to 4K/n (Thm 4.1), plus binned bias ≤ K/n with a matching tightness example (Lemma 4.1, Remark 4.1: ≥ (1−1/e)K/6n under perfect calibration).
Honest boundary analysis
Practical tests with a large efficiency edge
Rigor on ties and reproducibility
Weaknesses
One-sided measure with an unverifiable fidelity condition
Approximation baselines limited to binned ECE
The measure is a sequence, not a functional
Binary-only with ad-hoc multiclass handling
Modest evaluation resolution for test calibration
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.13100) asks how to measure miscalibration of a binary probabilistic classifier when the target, ℓ2-ECE(f) = E[(E[Y|f(X)] − f(X))²], cannot itself be estimated with guaranteed accuracy: Remark 3.2 (adapted from Angelopoulos et al. 2024, full proof in Appendix D.9) restates that no uniformly consistent estimator of ℓ2-ECE exists. The proposal is rankECE (§2): sort the predicted probabilities Z = f(X) and average residual products (Y−Z)(Y−Z') over consecutive order statistics — a sample analogue of the identity ℓ2-ECE = E[(Y−Z)(Y′−Z)]. Headline claims: rankECE is tuning-free and estimable without assumptions, characterizes calibration exactly, and approximates ℓ2-ECE at least as well as binned ECE.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is rankECE and how is it different from the usual ECE?
rankECE is a measure of miscalibration for binary classifiers. It sorts the predicted probabilities and averages products of residuals (Y − Z) from neighboring sorted points, motivated by the identity that ℓ2-ECE equals the expected residual product of two independent draws at the same predicted value. Unlike ECE, it can be estimated from finite samples with guarantees requiring no assumptions on the model, and it is tuning-free, whereas common binned ECE depends strongly on the number of bins and carries bias up to K/n with K bins.
Does a small rankECE prove my model is calibrated?
Population rankECE is exactly zero if and only if the model is perfectly calibrated (Theorem 3.1, n ≥ 4). However, rankECE is always a lower bound on ℓ2-ECE: if the residual function oscillates very rapidly (total variation growing with the sample size), rankECE can be near zero while true ECE stays positive (Appendix C). Small rankECE certifies absence of the miscalibration that rankECE can see, not uniform closeness to ECE.
Why is ECE hard to estimate in the first place?
The paper recalls (Remark 3.2, adapted from Angelopoulos et al. 2024) that for any estimator of ℓ2-ECE there exist distributions under which the expected error is at least c(1−c)/2, a gap that does not vanish as the sample size grows, so no estimator is uniformly consistent for ECE without assumptions; this is why proxies like binned ECE or rankECE matter.
Is there code and data to reproduce the experiments?
Yes. The paper links a code repository (github.com/anirbanc96/rankece). Experiments use synthetic conditional probability functions and public data: the Amazon and Yelp Review Polarity corpora plus four publicly available Hugging Face sentiment models (DistilBERT SST2, BERT SST2, RoBERTa Twitter Sentiment, BERT Multilingual Stars).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text contains explicit '[...sampling gap...]' markers: the Discussion (§6), the full statements of the §3.2 tests (Prop 3.4, Thm 3.2, Cor 3.1) and part of the §5.2 power comparison are not visible in the excerpt; judgments about power rest on the authors' in-text claims.
- Plotted values in Figures 3-5 and A.2-C.7 are not recoverable from the extracted text (only axis labels and in-text descriptions); quantitative approximation-quality judgments rely on the authors' descriptions.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.