VFR-Audit: Verdict-Level Reliability for Fairness Audits in Hospital Length-of-Stay Prediction
Md Jannatul Rakib Joy, Viet Vo, Caslon Chua
cs.AI
Quick verdict
Proposes a bootstrap verdict-flip-rate for fairness audits; solid single-dataset study, but gains over plain threshold shifting are marginal and redistributed.
Summary
The paper proposes VFR-Audit, a framework for assessing the reliability of thresholded fairness verdicts in clinical AI, validated on hospital length-of-stay prediction. Its core instrument is the Verdict Flip Rate (VFR), a scalar in [0, 0.5] equal to the minority-verdict count over K=500 stratified bootstrap resamples of the audit cohort, complemented by audit-size sensitivity (minimum cohort size where metric CV falls below 5%) and cross-hospital verdict agreement (Fleiss' kappa over 20 hospital-grouped folds), combined into a four-tier per-cell reliability classification. On Texas-100X (925,128 discharge records from 441 Texas hospitals; task: LOS > 3 days), 146 of 336 audit cells flip verdict at least once on the unintervened XGBoost model, and 11 of 28 cells still flip after an intervention that passes the four-fifths Disparate Impact rule on all four protected attributes. The authors report that intersectional reweighing never satisfies DI >= 0.80 regardless of regularization strength, while per-group threshold shifting is the decisive component; their greedy refinement improves mean VFR only marginally over plain threshold shifting (0.0863 to 0.0809) and redistributes residual instability from Age to Race. Main limitations: a single 2006 administrative dataset with non-standard race coding, threshold tuning on the audited partition in the headline split, stratified bootstrap freezing the demographic composition the paper says drifts, and no significance testing of the small between-baseline differences.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Point-estimate fairness passes do not imply verdict reliability: 146 of 336 audit cells flip at least once under K=500 stratified bootstrap on the unintervened panel, and 11 of 28 cells still flip after an intervention passing DI >= 0.80 on all four protected attributes.
- VFR, the minority-verdict proportion over bootstrap resamples bounded in [0, 0.5], converts metric-level uncertainty into a verdict-stability scalar suitable for governance reporting, with a fixed K=500 justified by worst-case binomial standard error.
- On Texas-100X, intersectional reweighing fails the four-fifths rule regardless of regularization strength (best Age-DI 0.283 over a 20-configuration sweep), while per-group threshold shifting is the component responsible for the all-four-DI pass.
- VFR-Audit's greedy refinement reduces mean VFR (0.0863→0.0809) and max VFR (0.490→0.476) versus plain threshold shifting at +0.36 pp accuracy, but redistributes residual instability from Age-axis to Race-axis cells.
Strengths
Verdict-level reframing matches the governance object
VFR (Eq. 1, §3.3) is bounded in [0, 0.5], reuses bootstrap machinery auditors already run, and §3.3.4 correctly positions it against bootstrap CIs, including the right-skew argument for ratio metrics like DI; the separation of fairness status from verdict reliability (stable-fair / stable-unfair / unstable) is operationally apt.
Honest negative and redistribution reporting
§4.2.3 explicitly shows the greedy refinement moves DI-Age VFR 0.412→0.232 but DI-Race 0.410→0.476 and SPD-Race 0.398→0.470, and warns that reporting only the 6.3% aggregate mean-VFR gain would conceal the redistribution; reweighing's failure is given a mechanism (base-rate floor: Elderly 0.606 vs Young Adult 0.207, 2.93×), not just an observation.
Layered robustness checks
Scale, multi-site structure, and release
Weaknesses
Headline gain over threshold shifting is marginal and untested
Stratified bootstrap freezes the drift the paper motivates with
Headline protocol tunes on the audited partition
Axis 3 kappa construction is statistically fragile
External validity and arbitrary constants
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a real operational gap: hospital fairness audits reduce continuous metrics to binary pass/fail verdicts against thresholds (e.g., DI ≥ 0.80), and those verdicts may not reproduce across repeated audits. It proposes the Verdict Flip Rate (Eq. 1, §3.3): the minority-verdict proportion over K=500 stratified bootstrap resamples, bounded [0, 0.5], with K fixed by a worst-case binomial SE argument (±0.044 half-width) plus a Hoeffding bound. Around it sit three axes — resampling stability, audit-size sensitivity (N* where CV < 0.05, R=30), and cross-hospital agreement (Fleiss' κ over 20 GroupKFold folds) — combined into a conjunctive four-tier per-cell classification. Headline claims: 146/336 cells (43.5%) flip on the unintervened twelve-model panel; 11/28 still flip after the all-four-DI-passing intervention; reweighing cannot reach DI ≥ 0.80 at any λ; threshold shifting is the effective component; and VFR-Audit modestly improves reliability over plain threshold shifting. Scope is a single retrospective administrative cohort (THCIC FY2006); §3.3.3 correctly frames VFR values as cohort-and-protocol-bound estimates, not prospective guarantees.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Verdict Flip Rate (VFR) in this paper?
VFR is a number between 0 and 0.5 measuring how often a binary fairness verdict (pass/fail against a threshold such as Disparate Impact >= 0.80) reverses across 500 stratified bootstrap resamples of an audit cohort. It equals the minority-outcome count divided by 500: VFR = 0 means every resample agrees, VFR = 0.5 means a 50/50 split. The paper treats VFR <= 0.10 as practically stable.
What data does the VFR-Audit paper use?
Texas-100X: 925,128 inpatient discharge records from 441 Texas hospitals, drawn from the THCIC Public Use Data File for fiscal year 2006. The prediction task is whether length of stay exceeds three days (45.0% positive rate), with race, sex, ethnicity, and age as protected attributes.
What are the main findings of the paper?
On the unintervened XGBoost model, 146 of 336 (model, metric, attribute) audit cells flip verdict at least once under resampling. After a per-group threshold-shifting intervention that passes the four-fifths Disparate Impact rule on all four protected attributes, 11 of 28 cells still flip. Intersectional reweighing never reaches DI >= 0.80 on this cohort regardless of regularization strength.
Is code or data available for VFR-Audit?
Yes. Code and experiment notebooks are at github.com/MdJoy31/VFR-Framework, and the underlying THCIC PUDF 2006 file is publicly downloadable from the Texas Department of State Health Services.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 appear only as captions in the extracted text; plotted per-cell values (e.g., the Figure 3 VFR heatmaps and Figure 4 fold distributions) could not be independently verified.
- No truncation detected: the extraction ends with the complete reference list through [72].
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.