How Humans and LLMs Read Gender into Gender-Neutral Physical Descriptions
Yingjia Wan, Lin Lin, Elisa Kreiss
cs.CL · cs.AI · cs.CY · cs.HC
Quick verdict
GAPA shows physical descriptions carry graded gender associations; LLM audit finds compression and non-binary-skewed abstention; solid, minor reporting gaps.
Summary
The paper tests whether the fairness practice of replacing gender labels with 'objective' physical descriptions actually yields gender-neutral communication. The authors build GAPA: 316 physical attributes drawn from LLM generation, human elicitation, and six novels, normed with 14,706 seven-point ratings from 304 US annotators (each attribute rated for only one target gender, ~17 ratings per attribute-gender cell). 53% of attributes are significantly more associated with one gender; associations are most consistent for women and men and weakest for non-binary targets. Evaluating 16 LLMs zero-shot against these norms, models only partially recover human associations (best: Claude-Opus-4.6, r=0.669; worst: GPT-OSS-20B, r=0.173), compress ratings toward the scale midpoint, align worse for the man category, and abstain disproportionately on non-binary prompts — abstention concentrated in instruction-tuned and proprietary models. A LoRA-fine-tuned OLMo2-7B proxy predictor reaches r=0.764 on the held-out LLM-generated test split and is applied to LitBank character descriptions, where predicted associations track character gender and, for male characters, author gender. Limitations: ratings capture socially shared stereotypes rather than inherent meaning; attributes are judged in isolation; annotators are US-only; the abstention metric conflates bottom-scale answers with refusal; and reported counts are internally inconsistent (316 vs 315 attributes; 152 vs 75 female participants between main text and appendix).
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Physical attribute descriptions carry structured, graded gender associations among human readers; 53% of the 316 GAPA attributes are significantly more associated with one gender than the others.
- Associations are more consistent and distinctive for women and men than for non-binary identities, structuring the semantic space around a binary axis.
- 16 LLMs partially recover human associations but compress ratings toward the scale midpoint, align worse for the man category, and abstain disproportionately on non-binary prompts, especially instruction-tuned and proprietary models.
- A fine-tuned OLMo2-7B proxy predictor (r=0.764 on the held-out in-domain test) enables scaled analysis, and its application to LitBank suggests character descriptions are gendered by both character gender and author gender.
Strengths
Novel, consequential research question with first large-scale evidence
The paper empirically tests an assumption (descriptions = neutral) that underpins accessibility/fairness guidance (Bennett et al. 2021; Scheuerman et al. 2019/2020, Sec. 1-2); prior work (Gao & Kreiss 2025) is acknowledged as only a first exploration.
Carefully designed human norming at scale
14,706 ratings from 304 annotators; each participant rates an attribute for one target gender only (Sec. 4.1), minimizing carryover and cross-gender comparison; counterbalancing, attention checks (≥4/5), and demographic covariates are reported.
Principled treatment of subjective ground truth
Broad model coverage with elicitation cross-checks
Released artifacts enable scaled research
Weaknesses
Abstention metric conflates valid low ratings with refusal
Internal inconsistencies in sample and counts
No multiple-comparison control for attribute-level tests
Single prompt template, no sensitivity analysis
Predictor's out-of-domain evidence not surfaced
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a concrete, policy-relevant assumption: that replacing identity labels ("she", "his") with physical descriptions ("short hair") yields gender-neutral communication, as recommended in accessibility and fairness work (Bennett et al., 2021; Scheuerman et al., 2019/2020). Three claims are advanced: (1) physical attribute descriptions carry structured, graded gender associations among human readers — 53% of 316 GAPA attributes are significantly more associated with one gender; (2) 16 LLMs only partially recover these associations, compress ratings toward the scale midpoint, align worse for the "man" category, and abstain disproportionately on non-binary prompts; (3) a released fine-tuned proxy predictor (r=0.764) enables scaled analysis, demonstrated on LitBank character descriptions. The "first empirical evidence" framing slightly overreaches given the authors' own citation of Gao & Kreiss (2025) as a first exploration, but the scale and multi-source design here are clearly novel.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do physical descriptions like 'a chiseled jawline' work as gender-neutral alternatives to gender labels?
According to this COLM 2026 study, no: in the GAPA dataset, 53% of 316 physical attributes were rated by 304 US annotators as significantly more associated with one gender than others, so descriptive language retains systematic gender associations rather than neutralizing them.
What is the GAPA dataset?
GAPA (Gender Associations of Physical Attributes) contains 316 physical attributes collected from LLM generation, human elicitation, and six novels, with 14,706 seven-point ratings from 304 US-based annotators on how strongly each attribute is associated with women, men, and non-binary people.
How well do LLMs match human gender associations of physical descriptions?
Across 16 LLMs evaluated zero-shot, correlations with averaged human ratings ranged from r=0.173 (GPT-OSS-20B) to r=0.669 (Claude-Opus-4.6). Models compress ratings toward the middle of the scale, align less well for the man category, and abstain disproportionately on non-binary prompts, mainly in instruction-tuned and proprietary models.
Is the GAPA predictor model publicly available?
Yes. The authors release a fine-tuned OLMo2-7B regression model that predicts human gender-association ratings (r=0.764, RMSE=0.633 on the held-out LLM-generated test split) on Hugging Face, alongside the dataset and evaluation code on GitHub.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels (Figs. 3, 5, 6, 8) were extracted as garbled text runs; only values quoted in prose or tables could be verified.
- Table 3 shows roughly nine of the 16 evaluated models; several proprietary models' per-source numbers are not visible in the extraction.
- One LitBank statistic is truncated mid-sentence ('vs. 0.71, p = .20'), so the character-gender magnitude comparison could not be fully verified.
- Appendix B.1 demographics may describe the separate image-elicitation study rather than the norming sample; the text does not clarify.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.