Taking the Whys Seriously: Limitations of Counterfactual Explanations in Justification and Recourse
Mattia Cerrato, Otto Sahlgren, Xenia Heilmann
cs.CY · cs.AI
Quick verdict
Argues upstream design choices reshape counterfactuals as much as method tuning; philosophically strong, empirically suggestive but reporting inconsistencies.
Summary
This paper asks whether counterfactual explanations (CEs) can legitimately justify algorithmic decisions and support recourse. Combining an empirical study with normative analysis, the authors cast a resume-screening pipeline as a structural causal model and intervene on 'upstream' design choices — label definition (salary caps $40k–$80k), feature measurement (four definitions of 'private-sector worker'), model-selection metric (accuracy, ROC-AUC, NDCG@100, precision, recall, F1), and validation strategy (holdout, 5-fold, 10-fold) — benchmarking their impact against tuning the explainer's validity threshold τ (0.51–0.70). Using US Census ACS microdata (folktables), random forests, and two retrieval-based generators (DiCE-kdtree, NICE) across ten seeds, they measure set distances and agreement scores (Jaccard, Sign, Value). Per the reported narrative, three of four upstream interventions move CEs beyond the τ band, and Value Agreement falls from 0.39 to 0.18–0.26 (DiCE-kdtree) and from 1.00 to 0.61–0.87 (NICE). The authors conclude that CEs presuppose contestable 'fixed model' choices, obfuscate organizational responsibility when offered as justification or recourse, and that recourse is relational. Limitations: no method-swap arm substantiates the abstract's comparison with 'the specifics of the generating method' (only τ is varied); internal inconsistencies (k=5/3 vs 4/2; 'three of four' versus NICE within-band for feature measurement; the conclusion citing 'a single CE method'); one dataset and model family; Student-t CIs only, without formal tests.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Upstream design choices — label definition (ϕY), feature measurement (ϕX), and model-selection metric (M) — perturb generated counterfactual sets more than the CE method's own validity threshold τ, while validation strategy (C) is roughly comparable.
- Upstream interventions produce at least as much counterfactual disagreement as τ tuning, with Value Agreement falling from 0.39 to 0.18–0.26 (DiCE-kdtree) and from 1.00 to 0.61–0.87 (NICE).
- CEs therefore rest on a contestable 'fixed model assumption' and, as putative justifications, conflate conditional model behavior with normative grounds while obscuring the decision-maker's responsibility.
- Algorithmic recourse is relational: CE-based recourse unilateralizes the burden of reversal onto subjects and, absent broader algorithmic transparency, risks steering them into complying with policies that may themselves require revision.
Strengths
Novel upstream-intervention framing with an explicit causal formalization
The pipeline is cast as an SCM (Fig. 1, Eqs. 2–8) whose design parameters ξ=(ϕX, ϕY, M, C, K, τ) become do-operator targets, turning 'business choices' into measurable interventions; Eq. 9 defines sensitivity against a downstream τ baseline.
Careful distance and agreement machinery
The set distance (Eqs. 10–11) is a symmetric mean-of-nearest-neighbors metric with MAD rescaling fitted per seed, and consistency extends Brughmans, Melis & Martens (2024)'s Jaccard with new Sign (continuous direction) and Value (categorical exact-value) agreements, aggregated over multi-CE sets (Eq. 16).
Honest calibration of scope and claims
Dual-method, dual-question empirical design
Substantive normative payoff tied to law and prior critique
Weaknesses
The 'generating method' comparison is asserted, not tested
Internal reporting inconsistencies
Confounded interventions
Narrow empirical scope and stretched proxy
Descriptive statistics only
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper (cs.CY/cs.AI, Aug 2026) couples an empirical sensitivity study with a normative argument about counterfactual explanations (CEs), targeting two high-stakes uses: justifying algorithmic decisions and providing algorithmic recourse (AR). Central claims: (1) interventions on 'upstream' pipeline choices — label definition ϕY, feature measurement ϕX, model-selection metric M — perturb generated CE sets at least as much as intervening on the explanation method's validity threshold τ (validation strategy C is comparable); (2) upstream interventions also increase counterfactual disagreement (Value Agreement: 0.39 → 0.18–0.26 for DiCE-kdtree; 1.00 → 0.61–0.87 for NICE, Fig. 3); (3) CEs therefore rest on a contestable 'fixed model assumption', cannot alone justify decisions, and in recourse wrongly localize the burden of reversal on decision subjects. The argument builds explicitly on Barocas/Selbst/Raghavan (2020), Kasirzadeh & Smart (2021), and Venkatasubramanian & Alfano (2020); the claimed empirical novelty is systematic quantification of *pipeline-level* sensitivity, complementing data/model-perturbation robustness work (Slack et al. 2021; Virgolin & Fracaros 2022).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What are counterfactual explanations and why do they matter for algorithmic recourse?
A counterfactual explanation tells a person the smallest change to their input data (e.g., income, work hours) that would flip an AI model's decision — for example, what a rejected loan or job applicant could change to be approved. They are widely proposed as a way to give rejected individuals actionable 'recourse'. This paper argues that such advice hides contestable upstream choices (how labels and features were defined, which metric selected the model), so it may point people toward complying with policies that themselves deserve challenge.
What did this paper find about counterfactual explanations?
Using US Census ACS data to simulate résumé screening, the authors changed 'upstream' pipeline choices — the salary-cap label rule, the definition of a private-sector worker, the model-selection metric, and the validation strategy — and measured how much the generated counterfactuals moved. Three of the four upstream changes reportedly moved explanations more than tuning the explainer's confidence threshold did, and agreement between explanations dropped sharply (e.g., categorical-value agreement from 0.39 to 0.18–0.26 for DiCE-kdtree, and from 1.00 to 0.61–0.87 for NICE).
Do counterfactual explanations satisfy the right to explanation under the GDPR or the EU AI Act?
The paper's position is that they are insufficient on their own. Counterfactuals can satisfy only a narrow reading of the GDPR's 'meaningful information about the logic' (Art. 21/22) and the AI Act's Art. 86 explanation rights, because they presuppose — but do not disclose — contestable design choices such as labeling rules and success metrics. The authors argue justification and recourse also require transparency about those upstream choices, not just a local explanation of the trained model.
What data and methods did the study use?
The experiments use American Community Survey microdata via the folktables library, treating each respondent as a job candidate with seven demographic/occupational features. 'Fit' is proxied by earnings (positive label = top-10% earner below a salary cap). Random-forest classifiers are trained under different metrics and validation schemes, and counterfactuals are generated with two open-source retrieval-based methods, DiCE-kdtree and NICE, across ten random seeds; no code release is mentioned in the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are present only as captions in the extraction; their plotted point estimates and CI widths are not recoverable, so all quantitative judgments rely on the in-text numeric descriptions.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.