A Fairness Audit of the Duckworth-Lewis-Stern Method: Format-Specific and Gender-Differential Bias, with an Interpretable Calibration Layer for Cricket Target Revision
cs.LG
Quick verdict
Audits DLS at scale (8,150 matches), finds an ODI gender bias gap; statistically careful, but practical impact of the ~6-run gap stays undemonstrated.
Summary
The paper presents the first large-scale empirical audit of the Duckworth-Lewis-Stern (DLS) target-revision method, using ball-by-ball data from 8,150 international matches (3,095 ODIs, 5,055 T20Is) sampled into 233,550 synthetic interruption scenarios with temporal train/validation/test splits. Two structured biases are documented: per-bucket mean DLS prediction error spans 137 runs across (overs-remaining, wickets-lost) states, and—novel to this paper—women's ODIs show systematically larger over-prediction than men's (+7.63 vs +1.51 runs on the training split; gap +6.13), confirmed under match-level block bootstrap, bucket-matched comparison (55/59 cells), Full-Member stratification, and a second DLS implementation; the T20I gap (+0.51) is not significant under clustered inference. The authors benchmark five ML alternatives and propose DLS-Cal, a 27K-parameter interpretable MLP adding a state-conditioned correction to DLS, cutting mean absolute bias 31% (ODI) and 19% (T20I); a gender-aware variant lowers women's test residual bias from +6.19 to +0.65 runs. On 417 real rain-affected matches the format-specific pattern persists, though that sample cannot resolve the gender gap. Limitations: synthetic interruptions mismatch real stoppage distributions, the proprietary ICC table forces a reconstruction, and the operational impact of a several-run bias (against RMSE near 40) is asserted rather than demonstrated; the Win-Flip Rate remains an unvalidated proxy for flipped outcomes.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- DLS prediction bias is structured rather than uniform: per-bucket mean error spans a 137-run range across (overs-remaining, wickets-lost) states, over-predicting T20 death overs and under-predicting ODI collapses.
- Women's ODIs show +6.13 runs larger mean DLS over-prediction than men's on the training split — claimed as the first quantification of gender-differential bias in DLS — robust to match-clustered inference, bucket matching (55/59 cells), Full-Member stratification, and implementation choice.
- DLS-Cal, a 27K-parameter interpretable residual MLP, reduces mean absolute bias by 31% (ODI) and 19% (T20I), and a gender-aware variant cuts women's test residual bias from +6.19 to +0.65 runs without degrading men's calibration.
- On 417 real rain-affected matches, DLS under-predicts ODIs (−11.1 runs) and over-predicts T20Is (+8.7 runs), corroborating the synthetic audit's format-specific pattern.
Strengths
First large-scale empirical DLS bias audit
8,150 internationals and 233,550 synthetic interruptions turn a long-theoretical critique (Bhattacharya et al. 2011) into quantified structure: per-bucket mean bias spans −111..+26 (ODI) and −19..+25 (T20I), a 137-run span (Section 3.2, Figure 1).
Unusually disciplined inference
Match-level block bootstrap (B=2,000) is primary because the ~30 samples/innings are correlated; ANOVA/Kruskal-Wallis (ODI F=527.16, T20I F=1,540.14) demoted to descriptive. The paper even retracts its own naive T20I Welch result (F=6.96, p=0.008) as a clustering artifact (Section 3.3).
Multi-pronged robustness of the gender finding
Honest scope framing and real-match validation
Interpretability backed by ablations
Weaknesses
Practical materiality not demonstrated
Abstract retains statistics the body disavows
Internal reporting inconsistencies
Overstated temporal stability and thin plateau evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper claims four firsts: a large-scale empirical audit of DLS prediction bias (8,150 internationals, 233,550 synthetic interruptions), the first quantification of gender-differential bias in DLS (+6.13-run ODI gap on the training split), an interpretable calibration layer (DLS-Cal, 27K parameters), and a thresholded fairness metric (Win-Flip Rate). The framing is unusually careful for an audit: Section 2.1 openly concedes that the study probes first-innings remaining-runs rather than operational two-innings target setting, arguing the two coincide exactly where DLS's proportionality assumption holds — a qualification many prior critiques lacked. Headline findings are (i) per-bucket mean bias spanning −111..+26 on ODI and −19..+25 on T20I (the combined 137-run span; §3.2), and (ii) the gender gap, which the abstract sells with F = 195.16, p < 10⁻⁴³.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What gender bias in the DLS method did this paper find?
In women's One Day Internationals, DLS over-predicts remaining runs by +7.63 on average versus +1.51 in men's ODIs at the same period — a +6.13-run gap. Because DLS uses one resource table for both genders, women chasing revised targets tend to receive targets several runs higher relative to their scoring environment. The gap survives match-level bootstrap, bucket-matched comparison, and top-nation-only subsets; in T20Is the gap (+0.51) is not statistically significant.
What is DLS-Cal and how much does it improve DLS?
DLS-Cal is a small (27K-parameter) neural network that adds a state-dependent correction to the standard DLS prediction. On the held-out test set it reduces mean absolute bias by 31% for ODIs and 19% for T20Is, and a version trained separately per gender lowers women's ODI residual bias from +6.19 to +0.65 runs while leaving men's calibration essentially unchanged.
Is the DLS gender bias large enough to change match results?
The paper shows the bias is statistically robust but does not demonstrate outcome impact: individual predictions carry RMSE of roughly 25–43 runs, and the proposed Win-Flip Rate is an error-threshold proxy explicitly not yet validated against actual flipped results. A several-run systematic bias plausibly matters in close chases, but outcome-level evidence is left to future work.
Can the study be reproduced?
The paper states that all input data is public (Cricsheet ball-by-ball) and that code, trained checkpoints, training logs, and the audit dataset are released on GitHub (sami42200/dls-cal), with a fixed random seed and fully documented hyperparameters and temporal splits.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 (bias heatmaps, correction structure, reliability diagrams) are referenced but their graphical content cannot be assessed from the extracted text.
- The released GitHub repository's actual contents (code/checkpoints/data) could not be verified offline; availability is reported as stated in the paper.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.