Counterfactual Fairness Audits of Multi-Step Clinical LLM Agents Require a Measured Per-Action Instability Floor
Rohith Reddy Bellibatlu, Manpreet Singh, Deepak Parashar, Rahul Joshi
cs.CL · cs.LG
Quick verdict
Claims fairness audits need a measured per-action instability floor; rigorous, unusually candid, but only 16 vignettes, 2 models, no disparity results yet.
Summary
This paper argues that counterfactual fairness audits of clinical LLM agents are uninterpretable without a measured per-action instability floor. Re-running an identical condition ten times over sixteen synthetic vignettes, the authors found the agent's action changed in 8.7% of outcome-vignette cells (374/4320 comparisons), with instability varying eightfold across six actions — from 0.022 for ICU escalation to 0.179 for a controlled-substance caution flag. Heterogeneity survives prevalence adjustment. A second model yields a pooled floor of 6.7% and nearly identical per-action ranking (Spearman 0.94, exact permutation p=0.017). Majority-vote aggregation over five replicates removes 39% of the floor and then plateaus; a null simulation attributes the residue to heterogeneous per-cell rates, so no deterministic-decode floor is estimable. No demographic contrast in a four-vignette pilot exceeded the floor, and no disparity is claimed. The measurements come from FairMedAgent, a released six-stage clinical-agent harness whose headline estimand — a within-range counterfactual flip rate that counts only flips between clinician-adjudicated acceptable actions — remains undefined because band adjudication is incomplete: one non-author geriatrician reviewed draft bands, no triage nurse or interpreter has, and no inter-rater agreement is computable. Key limitations: the floor study uses sixteen vignettes, provider-default sampling (temperature-zero floor is not measurable), and only two models; the rank correlation is fragile at six actions. Strengths include exceptional candor about negative and self-refuting results, prevalence-adjusted heterogeneity analysis, a concrete four-step floor protocol, and small-cluster-appropriate statistics.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Counterfactual flip rates from clinical LLM agent audits are uninterpretable without a measured per-action instability floor reported beside them.
- The instability floor is substantial (8.7% pooled, range 0.022–0.179 across six actions) and heterogeneous, and its heterogeneity survives prevalence adjustment.
- The floor is not model-specific in level (6.7% for a second model) and its per-action ordering is nearly identical across models (Spearman 0.94, exact p=0.017).
- Majority-vote aggregation removes only 39% of the floor and plateaus; the residue is fully consistent with heterogeneous per-cell rates, so neither decoding pinning nor replication demonstrably eliminates it.
- A within-range counterfactual flip rate conditioning on clinician-adjudicated acceptable-action bands separates demographic sensitivity from clinical error, and is the appropriate headline estimand for agent fairness audits.
Strengths
The core measurement directly supports the thesis
Sec. V-A quantifies an identical-condition floor of 8.7% (374/4320) with a percentile cluster bootstrap CI [0.056, 0.116], then decomposes it per action (Table II: 0.022 ICU escalation to 0.179 controlled-substance caution) and shows the eightfold spread survives prevalence adjustment (ratios 0.05–0.42 vs. a homogeneous-rate null at 2p̄(1−p̄)).
Exceptional candor, including self-refuting results
The paper reports that its boundary-proximity mechanism reversed at sixteen vignettes (Sec. V-B), that its four-vignette pilot floor of 0.136 was an artifact, and that two apparently significant analgesia cells in Sec. V-C were disqualified by a prompt asymmetry the authors introduced themselves. This is rare and strengthens trust in the remaining claims.
Cross-model replication with exact inference
Concrete, adoptable protocol
Sound small-cluster statistics and preregistration discipline
Weaknesses
The floor evidence base is very small
The headline estimand is never exercised on data
The floor was measured under uncontrolled decoding
Generalization is asserted on thin rank evidence
Underspecified weakest action dilutes the headline number
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a measurement-methodology paper, not a disparity study — and it says so repeatedly. The claim structure is narrow and disciplined: (1) counterfactual flip rates from clinical-agent fairness audits are uninterpretable without a measured per-action instability floor; (2) the floor is large (pooled 8.7%, 374/4320, CI [0.056, 0.116]) and heterogeneous eightfold across six actions (Table II); (3) it is not one model's artifact (second model 6.7%, Spearman 0.94); (4) replication mitigates without eliminating (−39% at R=5, then plateau, null-simulation-consistent); and (5) FairMedAgent supplies a within-range estimand that separates demographic sensitivity from clinical error — *specified but not exercised*, pending band adjudication. The abstract's headline sentence — 'Any counterfactual fairness estimate reported without a per-action floor beside it cannot be read as evidence of disparity' — is the actual contribution, and the data support it at the demonstrated scale.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is an instability floor in LLM fairness auditing?
It is how often an LLM agent gives a different answer when you re-run the exact same input, changing nothing. This paper measured 8.7% for a clinical agent over 4320 comparisons — meaning roughly one in twelve identical-condition comparisons flipped. Any demographic difference smaller than this floor could just be decoding noise, so fairness audits should report the floor next to every contrast.
Did this paper find that clinical AI is biased against certain demographic groups?
No. The paper explicitly claims no disparity results. Its four-vignette pilot found no demographic contrast distinguishable from the model's self-instability. The paper is about measurement methodology: it argues that published flip-rate audits without a noise floor cannot be interpreted, and its own bias-measuring metric (within-range flip rate) awaits clinician band adjudication.
Why can't you just set temperature to zero to make LLM outputs deterministic?
Temperature controls the sampling step, but the underlying logits still vary with hardware, batch composition, and floating-point reduction order. The paper cites work showing up to nine-point accuracy variation under fixed sampling. So greedy decoding removes one noise source but not all, and hosted-API users cannot control batch composition — the residual floor must be measured, not assumed away.
Is the FairMedAgent code and data available?
The evaluation harness (v0.1.1) is on GitHub (rohithreddybc/FairMedAgent) and archived on Zenodo (10.5281/zenodo.22165979); the raw trajectories behind every reported number are on Hugging Face (Rohithreddybc/FairMedAgent). The development split, datasheet, and leaderboard are promised only after clinician validation completes; the test split remains sealed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The text appears complete through the references ([53], acknowledgment) with no mid-sentence cutoff; no truncation is claimed.
- The full pilot tables, supplement, and TRIPOD-LLM checklist are referenced but not included in the provided text; judgments on them rest on the main-text descriptions only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.