Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicity
Sinjini Banerjee, Tim Marrinan, Anand D. Sarwate
stat.ML · cs.AI · cs.LG
Quick verdict
New margin+variability consistency measure audits Rashomon-set ensembles with concentration guarantees; theory conservative, empirical edge over margin small.
Summary
The paper studies predictive multiplicity—the Rashomon effect, where equally accurate models disagree on individual inputs—through the lens of auditing systems that divert uncertain predictions to human reviewers. The authors propose (β,σ)-consistency, a per-instance score combining an ensemble's margin (distance of the averaged Rashomon-set prediction from the 0.5 decision boundary) with a penalty for local prediction variability, estimated by averaging output changes under random perturbations of radius σ in embedding space. Theoretically, under i.i.d. sampling of ensembles from the Rashomon set plus local Lipschitzness and stability assumptions, Theorem 1 shows finite-ensemble scores concentrate around the full-set (warehouse) values, with failure probability decaying exponentially in ensemble size M and perturbation count S. Empirically, using warehouses of T=100 fine-tuned models (BERT/RoBERTa on SST-2 and MRPC; T0+IA3 on Adult and Bank; ϵ=0.03 tolerance; 500 sampled ensembles), thresholding the score yields favorable audit risk/cost trade-offs: on MRPC/BERT, agreement with warehouse audit decisions rises from 73.1% (M=1) to 90.9% (M=30), and the score correlates strongly with multiplicity metrics (Spearman 0.964 with prediction variance at M=10), outperforming the confidence-based measure of Hamman et al. (2025). Limitations: binary classification only; the reference standard is itself a finite warehouse ensemble; a margin-only ablation correlates nearly as well; values of S, α, and any code link are not evident in the provided text.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- (β,σ)-consistency — an ensemble margin penalized by local perturbation variability — identifies instances whose predictions remain stable across ensembles sampled from a Rashomon set.
- Under i.i.d. retail sampling, local Lipschitzness and ϵ2-stability, Theorem 1 shows finite-ensemble consistency scores concentrate around the warehouse values with failure probability decaying exponentially in M and S.
- Thresholding the measure for diversion substantially reduces audit risk versus single-model auditing at only moderate diversion increases, and retail ensembles of modest size (around M=10) approximate the full set's audit behavior.
- The proposed measure correlates more strongly with established multiplicity metrics than prior consistency measures, including the confidence-based score of Hamman et al. (2025).
Strengths
Well-designed auditing formalization
Table 1's 4×4 candidate-vs-baseline outcome matrix cleanly decomposes divergence into excess audit cost/risk versus the baseline's own unavoidable errors, and Definition 7's four metrics (ratio_cost, ratio_risk, frac_div, frac_risk) make the diversion trade-off measurable; the operating-point selection procedure in Section 4.2.2 is fully specified.
Clean, honest concentration theory
Theorem 1 (proven in A.3 via McDiarmid, Lemma 3, and Hoeffding-based Lemmas 4–5) yields failure probability decaying exponentially in both ensemble size M and perturbation count S; assumptions (i.i.d. retail sampling, η-local Lipschitzness, ϵ2-stability) are explicit, and the paper candidly notes the guarantee is silent for x ∉ Dcons_test.
Two-modality empirical validation matched to the theory
Sharp diagnosis of the prior measure
Weaknesses
Empirical edge over its own margin-only ablation is thin
Reference standard is a finite artifact
Audit threshold appears tuned on test data
Theory–practice gap
Narrow task scope and unquantified deployment cost
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses predictive multiplicity — the Rashomon effect, where near-equally-accurate models disagree on individual inputs — from the operational angle of auditing systems that decide whether a prediction should be automated or diverted to a human expert. The central objects are a *warehouse* (the expected model over an empirical Rashomon set) and *retail* ensembles: finite i.i.d. draws of M models. The proposed (β,σ)-consistency score (Definitions 3 and 6) combines an ensemble margin, f̄(R,x) − 1/2, with a penalty term: the average change in member-model outputs between x and S uniform perturbations of radius σ on the embedding-space sphere. Three claims are advanced: (1) a concentration guarantee (Theorem 1) that retail scores approach warehouse values with probability decaying exponentially in M and S; (2) thresholding the score yields favorable audit risk/cost trade-offs; (3) the score correlates better with standard multiplicity metrics than prior consistency measures, notably the confidence-based hcσ of Hamman et al. (2025). Scope is deliberately narrow: probabilistic binary classification with uniformly weighted ensembles.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What are the Rashomon effect and predictive multiplicity?
The Rashomon effect is the observation that many distinct models trained on the same data achieve nearly identical accuracy yet disagree on individual predictions; predictive multiplicity is the resulting per-instance instability. This paper studies how ensembles drawn from such near-optimal model sets behave when used to audit predictions and divert uncertain cases to human reviewers.
What does the (β,σ)-consistency score measure?
For a test instance, it subtracts from the ensemble's margin (distance of the averaged prediction from the 0.5 decision boundary) a penalty equal to the average change in model outputs when the input embedding is perturbed by random noise of radius σ. High scores indicate predictions that are both far from the decision boundary and locally stable across ensemble members.
Did ensembling help in the experiments?
Yes. On MRPC with fine-tuned BERT, agreement between a sampled ensemble's audit decisions and the 100-model reference rose from 73.1% with a single model to 90.9% with 30 models, and consistency scores reached Spearman correlations up to about 0.96 with prediction-variance multiplicity metrics, outperforming a prior confidence-based measure.
What are the main limitations of this paper?
The framework covers binary classification only; the reference standard is itself a finite 100-model ensemble; a margin-only ablation performs nearly as well in the correlation tables; and the audit threshold appears to be tuned on test data, so the reported operating points may be optimistic.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with marked gaps; the concrete values of S (perturbation count) and α (radius scale) and other per-dataset hyperparameters likely sit in skipped sections, so 'not reported' judgments apply only to the visible text.
- Algorithm 1 and Definition 1 are referenced (e.g., Figure 2 caption, Section 4.1.1) but their content falls within the sampling gaps.
- In Table 2b (SST2), the consistency-measure row labels were lost in extraction; the measure identities there are inferred from Table 2a's structure.
- No code or data availability statement appears anywhere in the provided text; one may exist within the skipped sections.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.