LLMs Mirror Country-Specific Gender Patterns If Asked, but Skew Male When Generating Media in Local Languages
Sharif Kazemi, Tanya Popli, Neil K. R. Sehgal, Sunny Rai, Niyati Malhotra, Victor Orozco-Olvera, Ana María Muñoz Boudet, Samuel P. Fraiberger, et al.
cs.CL
Quick verdict
Four-country survey + 8-LLM audit: media generation skews male vs explicit elicitation; rigorous stats but Kenya/Nigeria baselines rest on one survey.
Summary
The paper asks whether LLMs reproduce country-specific gender stereotypes when generating media, and whether this differs from what models report when asked directly. The authors ran an original Facebook-Messenger survey of 695 respondents in the US, India, Kenya, and Nigeria, eliciting gender associations for 22 occupational and domestic roles, validated against US BLS (r=+0.95), India's PLFS formal sector (r=+0.89), and household-task benchmarks (r=+0.94). Eight LLMs — GPT-5.1, four generic open-weight models, and three locally fine-tuned models — were probed under explicit (direct question) and implicit (media generation) elicitation in English, Hindi, Swahili, Hausa, and Yoruba. Under explicit elicitation, models track surveyed associations (mean Pearson r=+0.81); under implicit media generation, alignment falls and outputs shift male (β=−0.72, 95% CI [−1.10,−0.34], n=37,707), concentrated in Hindi (β=−1.36) and Yoruba (β=−1.27) cells while English-prompted non-US cells show small, non-significant shifts. Implicit male share averages 66%, approaching the ~75% documented in film/TV content analyses. Instruction-based mitigations overshoot toward female, collapsing GPT-5.1's survey alignment from r=+0.45 to r=−0.22. Stated limitations include reliance on GPT-4o-mini as judge, a coarse ternary implicit scale, absent external validation for Kenya and Nigeria, a 30-person US sample, and Facebook-recruited sample skew.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under direct questioning, the eight evaluated LLMs broadly track country-specific surveyed gender associations (mean Pearson r=+0.81 across the five generic models and four countries).
- Switching from explicit questioning to media generation shifts outputs toward male protagonists (β=−0.72, 95% CI [−1.10,−0.34], n=37,707), with the shift concentrated in Hindi and Yoruba prompting and small and non-significant under English outside the US.
- Implicit-elicitation male shares (66% average across 22 roles) converge with the ~75% male over-representation documented in film/TV content analyses.
- Prompt-level mitigations are brittle: balanced/debias instructions overshoot toward female and collapse GPT-5.1's survey alignment from r=+0.45 to r=−0.22 and −0.09.
Strengths
Original human baseline where none existed
A 695-respondent four-country survey of 22 roles (§3.1), validated where statistics allow: US survey vs BLS r=+0.95, Indian formal-sector subset vs PLFS r=+0.89 (Fig. 6), and domestic tasks vs Pew/BLS/Gallup/DHS at r=+0.94 with 100% sign agreement on n=8 pairs (App. A.4).
Statistical discipline unusual for audit papers
Pre-specified primary estimand (§4.2), BH-FDR q-values for per-model and per-cell families (Tables 8-9), wild cluster bootstrap for the 22 task clusters (p=.0018, App. E.1), a scale-support collapse check (App. E.2), and four-way judge substitution preserving the headline sign (Table 6: β from −0.719 to −0.418).
Nulls reported and interrogated rather than buried
Prompt language treated as a design axis with formal interaction tests
Judge robustness documented at the row level
Weaknesses
Baseline validation is asymmetric across the countries that carry the headline
Single-judge measurement layer with a modest human reference
The regime contrast bundles format, scale and prompt structure
Small, skewed samples in pivotal cells
Mitigation arm lacks a placebo and leans on an atypical anchor
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: LLMs Mirror Country-Specific Gender Patterns If Asked, but Skew Male When Generating Media in Local Languages
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do LLMs reinforce gender stereotypes when generating stories or media?
According to this study, yes: across eight tested LLMs, switching from direct questioning to media generation shifted outputs toward male protagonists by over half a Likert step (β=−0.72 on a [−2,+2] scale). Average implicit male share was 66% of protagonist roles, close to the ~75% male over-representation documented in human-made film and television.
Does testing LLMs only in English hide gender bias in other countries?
The paper argues it can. Outside the US, implicit male shifts were small and statistically non-significant when prompting in English, but local-language prompting produced 4–6× larger shifts in Hindi and Yoruba and roughly double in Swahili. The authors conclude English-only or country-agnostic audits would miss bias in the languages where these models are actually deployed.
Can you fix LLM gender bias with a prompt instruction?
Not reliably, per this study. Instructions to balance representation or avoid defaulting to a male protagonist reduced the male shift directionally but overshot toward female, collapsing GPT-5.1's alignment with local survey data from r=+0.45 to as low as −0.22, while a realism instruction produced no reliable improvement.
Which language models were evaluated in this study?
Eight models: GPT-5.1 (frontier closed), LLaMA 4 Scout, LLaMA 3.1 8B, LLaMA 3.0 8B, Mistral Small 24B (generic open-weight), and three locally fine-tuned models — Sarvam-M for India (Hindi), UlizaLlama3 for Kenya (Swahili), and N-ATLaS for Nigeria (Hausa/Yoruba).
How was the human baseline for gender associations collected?
The authors surveyed 695 adults (India 205, Kenya 260, Nigeria 200, US 30) recruited via country-targeted Facebook ads, interviewed through a Facebook Messenger bot in English or local languages. Respondents rated 22 occupational and domestic roles on a five-point scale from 'almost always done by men' to 'almost always done by women'. The US version matches official labor statistics at r=+0.95.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-9 carry quantitative content (e.g., per-cell coefficients and CIs in Figure 4, per-role shares in Figure 3) that is not recoverable from text extraction; numeric claims in this review were verified against the tables and in-text values only.
- Appendix B.3 cross-references §8 for a Yoruba translation error and its data treatment, but no such discussion appears in the extracted §8 — treated here as a cross-reference inconsistency rather than assumed truncation, since the document ends cleanly at Appendix E.2.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.