WinoQueer-NL: Assessing Bias in Dutch Language Models toward LGBTQ+ Identities
Jiska Beuk, Gerasimos Spanakis
cs.CL
Quick verdict
Dutch WinoQueer adaptation with queer community survey finds trans/non-binary model bias up to ~97% despite neutral averages; valuable resource, thin stats.
Summary
This paper introduces WinoQueer-NL, a Dutch adaptation of the English WinoQueer benchmark for measuring anti-LGBTQ+ bias in language models. The authors translated the benchmark's components (templates, names, identities, 177 predicates), dropped six problematic translations, and validated cultural relevance through an online survey of 43 Dutch-speaking queer participants, who confirmed 145 of 171 stereotypes and contributed 22 new Dutch-specific ones. The released dataset contains 42,906 stereotypical/counter-stereotypical sentence pairs across nine identities. Thirteen models — Dutch and multilingual masked and autoregressive LMs — were scored by the percentage of pairs where the stereotypical sentence receives higher log-likelihood. Aggregate scores hover near the neutral 50% mark (mean 50.53%), but identity-level analysis reveals extreme disparities: GPT2 Dutch large favors stereotypical sentences for transgender identities 96.80% of the time and GPT2 Dutch small reaches 98.33% for non-binary, while Llama-3.2-1B scores only 5.85% for gay-related pairs, which the authors interpret as possible reverse bias. Transgender and non-binary identities show consistently elevated bias across most models. A Pearson correlation analysis finds stereotype harmfulness does not predict model bias (|r| < 0.12, r² ≤ 1.4%). Limitations include the small survey sample (6–9 ratings per stereotype; only 6 raters for new stereotypes, deemed relevant by default), no confidence intervals or significance tests on bias proportions, and non-comparable MLM/ARLM scoring schemes.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A culturally adapted Dutch benchmark (WinoQueer-NL) with 42,906 sentence pairs and 167 community-validated stereotypes enables measurement of anti-LGBTQ+ bias in Dutch and multilingual models.
- Aggregate bias scores appear near-neutral (mean 50.53%) but conceal extreme identity-level disparities, with transgender and non-binary identities consistently most biased (up to 96.80% and 98.33% for individual models).
- Usage-weighted bias is lower than identity-averaged bias for most models, showing that overrepresented identities dilute measured bias.
- Stereotype harmfulness ratings do not predict the magnitude of model bias (|r| < 0.12, r² ≤ 1.4% across 13 models).
- Very low bias scores (e.g., 5.85% for gay-related pairs) may indicate reverse bias toward non-marginalized groups rather than fairness.
Strengths
Community-validated cultural adaptation
43 Dutch queer participants confirmed 145/171 translated stereotypes and contributed 22 Dutch-specific ones; 26 irrelevant items were excluded via a transparent tier protocol (Section 3, Tables 3-4) rather than importing US-centric stereotypes wholesale.
Broad, current model coverage
13 models spanning Dutch-specific (BERTje, RobBERT, GPT-2 NL small/large, Fietje-2, Schaapje 2B) and multilingual (XLM-R, Llama-3.2-1B/3B, Phi-4-mini) architectures, both MLMs and ARLMs, sizes 109M-3.84B (Table 5).
Dual aggregation exposes score masking
Honest treatment of low scores
Credible null result on harmfulness
Weaknesses
No uncertainty quantification on headline scores
Validation asymmetry for new stereotypes
Cross-architecture comparability acknowledged then ignored
Unexamined tokenization and model-variant confounds
Causal overclaim in subset analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
WinoQueer-NL extends the English WinoQueer benchmark (Felkner et al., 2023) to Dutch, targeting anti-LGBTQ+ bias in language models. The paper claims three contributions: (1) a culturally adapted Dutch benchmark built from component-wise translation plus community validation, (2) evaluation of 13 Dutch-specific and multilingual models covering both masked (MLM) and autoregressive (ARLM) architectures, and (3) analysis of bias disparities across nine identity groups. The headline empirical claim is that aggregate bias appears neutral (mean usage-weighted score 50.53%, Table 6 bottom row) while identity-level scores diverge extremely: transgender bias reaches 96.80% (GPT2 Dutch large) and non-binary 98.33% (GPT2 Dutch small) on the full dataset, versus 5.85% for gay-related pairs (Llama-3.2-1B). On the 22 Dutch-only stereotypes (Table 11), extremes widen further — mBERT reaches 99.32% for transgender, exceeding the abstract's 'up to 97%'.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is WinoQueer-NL?
WinoQueer-NL is a Dutch-language benchmark for measuring anti-LGBTQ+ bias in language models, culturally adapted from the English WinoQueer benchmark. It contains 42,906 pairs of stereotypical and counter-stereotypical sentences built from 167 community-validated stereotypes, 9 LGBTQ+ identities, 60 Dutch names and 11 sentence templates. Its cultural relevance was validated through a survey of 43 Dutch-speaking queer participants, and it is publicly released on GitHub.
Which Dutch language models show the most anti-LGBTQ+ bias in this study?
Among the 13 models evaluated, GPT2 Dutch large scored highest on transgender-related pairs (96.80% preference for stereotypical sentences) and GPT2 Dutch small reached 98.33% for non-binary identities; RobBERT was the highest Dutch masked model (62.87% overall). Across most models, transgender and non-binary identities received the highest bias scores, while gay- and queer-related pairs often fell below the neutral 50% mark.
What does a bias score below 50% mean in WinoQueer-NL?
Scores above 50% mean the model more often finds the stereotypical sentence more likely than the counter-stereotypical one; scores below 50% mean it prefers the counter-stereotypical version. The authors caution that very low scores (e.g., 5.85% for gay-related pairs with Llama-3.2-1B) do not indicate fairness — they may signal reverse bias favoring the non-marginalized group.
Is the WinoQueer-NL dataset publicly available?
Yes. The paper states two datasets are released at github.com/jerryspan/WinoQueer-NL: the 42,906-pair sentence-pair dataset and a metadata file with per-stereotype relevance and harmfulness ratings collected in the community survey.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Equations 1-2 and sub/superscript notation (e.g., s<ui) are linearized by PDF extraction; they were reconstructed from surrounding prose and remain interpretable.
- Figures appear only as captions in the extraction (e.g., Figure 1, harmfulness distribution); all numeric tables (1-11) are fully present.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.