Verdict Instability of OOD Scores under Reference Resampling
cs.LG · stat.ML
Quick verdict
Parameter-free closed form for OOD verdict instability under reference resampling; well validated internally but vision-only and a circular abstention target.
Summary
The paper asks how much a post-hoc out-of-distribution (OOD) score would change if the reference set it is fitted on were redrawn. It defines verdict instability as the bootstrap standard deviation of the score under class-wise resampling of the reference set, and derives a parameter-free closed form: the within-class dispersion of the assigned class along the query's direction divided by the square root of that class's reference count, plus a rectified penalty channel for detectors with a global reference. The closed form tracks bootstrap variance with R2 of 0.82-0.97 across CIFAR-100, CIFAR-10 and imbalanced DermaMNIST (Table 1), and the 1/sqrt(nc) dependence is identified under natural imbalance up to 58.7x, where replacing nc by its mean collapses R2 from 0.923 to 0.276. Because far-OOD queries lie along low-variance directions of anisotropic embeddings, nine of eleven standard scores anti-correlate with verdict reproducibility; only dispersion estimators (knn std, LID) carry the expected sign. A label-free rule predicts each score's sign in all 55 score-setting combinations, and abstention driven by wrong-signed scores (e.g., Mahalanobis on DermaMNIST, 13.09% worse than random) underperforms random abstention. Main limitations: evidence confined to vision benchmarks with three frozen encoders, the closed form is derived for a proxy detector rather than the eleven scores themselves, and the abstention target is instability rather than downstream correctness.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The bootstrap standard deviation of a post-hoc OOD score under class-wise reference resampling admits a parameter-free closed form: transverse within-class dispersion divided by sqrt(nc) plus a rectified global-penalty channel (Eq. 1/7; Table 1: R2 0.82-0.97).
- The 1/sqrt(nc) dependence is identifiable only under class imbalance and is validated on DermaMNIST up to 58.7x, where replacing nc by its mean collapses R2 from 0.923 to 0.276 (Table 6).
- Far-OOD queries lie along low-variance directions of anisotropic embeddings (pooled rho(r, sigma_t) = -0.83, Table 2), so nine of eleven standard scores anti-correlate with verdict reproducibility and only dispersion estimators (knn std, LID) carry the expected sign.
- A single label-free correlation, sign of rho(M, T-hat), predicts the sign of every tested score's correlation with true instability across all 55 score-setting cells (Eq. 9; Appendices H-I).
- Abstention driven by a wrong-signed score is worse than random abstention in all tested cells (e.g., Mahalanobis on DermaMNIST: +13.09% +/- 0.16, Table 3), and which score fails changes per dataset exactly as the rule predicts (Section 6.3).
Strengths
Parameter-free closed form validated at scale
Eq. (7) matches the bootstrap ground truth with R2 0.820-0.974 and median T/T-hat 0.953-1.005 across four settings (Table 1), with nothing fitted and hyperparameters fixed before any instability was measured.
Clean identification of the reference-count term
Natural DermaMNIST imbalance (6.1x to 58.7x via a class-count threshold, Table 4) plus the single-substitution ablation nc -> mean-count (Table 6: R2 0.923 -> 0.276 at 58.7x) isolates the count's contribution; the exponent is read off classes with no tuning (Table 5, fixed-argmin gamma-c 0.97-1.03).
Falsification-grade controls
Above-norm statistical reporting
Honest scope statements
Weaknesses
Abstention target is partly self-referential
Theory-proxy gap
Narrow empirical scope
nc < d approximation regime
No code release stated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper studies post-hoc OOD detection through a lens the literature has largely ignored: the sampling variability of the detector itself. Because Mahalanobis, kNN and logit-family scores are all fitted on a finite in-distribution reference set R, every score is a statistic of R. The authors define verdict instability T(x) as the standard deviation of s(x; R*) over B=200 class-wise bootstrap replicates of R (Eq. 3), and make three claims: (1) T admits a closed form (Eq. 1/7) — transverse within-class dispersion sigma_t(u) = sqrt(u^T Sigma_c u) divided by sqrt(n_c), plus a rectified global-penalty channel — with no fitted parameters; (2) a label-free plug-in T-hat predicts the sign of any score's correlation with T (Eq. 9); (3) abstention on wrong-signed scores underperforms random abstention. All claims concern verdict reproducibility, not correctness — the paper is explicit that a confidently wrong detector can be perfectly stable (Section 3.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'verdict instability' in OOD detection?
It is the standard deviation of an OOD detector's score for a fixed input when the labeled reference set the detector was fitted on is resampled via the bootstrap. The paper shows this quantity has a closed form: the within-class spread of the assigned class along the query's direction divided by the square root of that class's reference count. A consequence is that far-OOD inputs often get the most reproducible verdicts, the opposite of what distance-based OOD scores implicitly assume.
Does this paper show that Mahalanobis or kNN OOD detectors are bad?
No. The paper's point is that detection quality and verdict reproducibility are different, nearly oppositely ordered targets. The scores that best separate in- from out-of-distribution inputs (MSP, Mahalanobis) are the worst at identifying which verdicts would flip under a different reference set, and vice versa (its Table 17). Each score is good at the job it was designed for and poor at the other.
Why can abstaining based on an OOD score be worse than abstaining at random?
Because some scores are anti-correlated with verdict instability: the queries they flag as 'uncertain' are precisely those whose verdicts are most reproducible. Deferring them leaves a retained set that is less stable than the starting population. For example, Mahalanobis-driven abstention on DermaMNIST in the paper is 13.09% worse than random abstention against the instability target.
Is code available for this paper?
No code release is mentioned anywhere in the paper text. Reimplementation is nonetheless straightforward: all datasets (CIFAR-10/100, DermaMNIST, standard OOD sets), the public timm DINO ViT-B/16 checkpoint, all detector hyperparameters, and seed protocols are specified, and compute is modest (one RTX 4090; the analysis loop runs on CPU in minutes once features are cached).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Equation typography (subscripts, radicals, hats) is partially garbled by PDF extraction, but all tables, values and appendix references A-M are legible.
- The provided text ends naturally at the close of Appendix M ('the choice is currently made by default.') with no sign of truncation; 'codeAvailable: false' reflects a verified absence of any code statement in the entire text, not a missing section.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.