Towards participatory speech dataset curation: A queer case study and conceptual framework
Brooklyn Sheppard, Anaelia Ovalle, Adina Williams, Levent Sagun
cs.AI · cs.CY
Quick verdict
Position paper: a four-phase participatory framework for queer speech data curation — timely and well-motivated, but unvalidated: no pilot, no evidence.
Summary
This position paper argues that speech dataset creation should adopt participatory methods from the outset, using the LGBTQIA+ (queer) community as a case study. The authors review why standard crowdsourced collection is poorly suited to queer speakers: gender annotations can effectively out non-binary and trans participants; documented AI harms against the community include 'gaydar' identification attempts, disproportionate toxicity flagging of queer dialects, and trans and non-binary users' deep distrust of voice-assistant developers. They survey prior participatory work — the PARQAIR-MH Delphi study in mental health, the StammerTalk stuttered-speech corpus, ethical desiderata for neurological speech data, and participatory turns in field linguistics — and distill a conceptual framework with four overlapping phases: defining the community (including who is excluded, e.g., the Global North and English skew of Queer in AI), project formulation (speech genres, public availability, storage, maintenance, and revocation), modes of participation (roles, collaboration, and acknowledgment via payment or authorship), and personal autonomy (individual choice versus group consensus), all connected by two-way knowledge sharing. The paper contains no experiments, dataset, or evaluation; the framework is offered as normative infrastructure rather than a validated protocol. Its main limitations are the absence of any pilot application, thin empirical grounding (the sole quantitative anchor is a Common Voice 16.0 analysis with 8–86 speakers per language), and unexamined novelty relative to existing datasheet-style checklists.
Review complete
Assessment at a glance
3
risk flags identified
● high 0
● medium 1
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Gender annotations in speech datasets, commonly treated as benign, function as sensitive markers that can out gender-diverse speakers, making standard crowdsourced collection unsuitable for queer participation.
- Existing participatory efforts each leave a gap: PARQAIR-MH allows consultation without downstream influence, StammerTalk retains a collector/contributor distinction, and Queer in AI skews toward the Global North and English speakers.
- A four-phase framework (community, project formulation, modes of participation, personal autonomy) connected by two-way knowledge sharing constitutes useful 'normative infrastructure' for community-led speech dataset curation.
- The framework generalizes beyond the queer case to other marginalized communities — a claim explicitly deferred to future work.
Strengths
Sharp problem framing: annotation-as-outing
§2.1 articulates that gender, the most routine annotation in speech datasets, becomes a sensitive category marker when gender-diverse speakers participate — arguably the paper's most original and consequential observation for dataset design.
Genuinely cross-disciplinary synthesis
§3 connects participatory/queer AI (PARQAIR-MH [24]), speech-specific curation (StammerTalk [26]), sensitive-data desiderata [25], Databrary governance [32], and field linguistics' two-way knowledge sharing ([27]–[29]) — the linguistics lineage is a distinctive import for the speech-AI audience.
Actionable guiding questions rather than abstract principles
Honest self-scoping and calibrated claims
Weaknesses
No validation or pilot application
Unclear novelty over existing documentation frameworks
Empirical motivation rests on third-party, tiny-sample evidence
Hard governance problems posed but unresolved
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a four-page position paper in Interspeech format (index terms: data, participatory, co-design, responsible, AI) by authors from the University of Calgary and Meta FAIR. Two contributions are declared in §1: (1) motivating participatory speech dataset curation through the lens of queer speakers, and (2) proposing a conceptual framework for curation 'by, for, and with' marginalized communities. The paper's sharpest idea appears in §2.1: gender — the most routine annotation in speech datasets — becomes a *sensitive* attribute when gender-diverse speakers are included, because volunteering it effectively outs the speaker. The paper claims that standard crowdsourcing pipelines structurally exclude or endanger queer participation; that prior participatory efforts each leave a gap (consultation without downstream power in PARQAIR-MH, §3; collector/contributor separation in StammerTalk, §3; Global North/English skew in Queer in AI, §4.1); and that a four-phase cyclical framework (community → project formulation → modes of participation → personal autonomy, §4) constitutes useful 'normative infrastructure'. A generalization claim (§5) extends it beyond the queer case.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 3 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does this paper present a new speech dataset or experiments?
No. It is a position/conceptual paper: it reviews speech collection practices and participatory precedents, then proposes a four-phase framework for community-led speech dataset curation. There is no dataset release, no model, and no quantitative evaluation; the only numbers cited come from prior studies, such as an ASR gender-gap analysis on Common Voice 16.0 involving 8–86 speakers per language.
What are the four phases of the proposed framework?
1) Community — who the project serves and who might be excluded; 2) Project formulation — speech genres, public availability, allowed uses, storage, maintenance, and revocation; 3) Modes of participation — roles, collaboration, and acknowledgment (payment, authorship); 4) Personal autonomy — balancing individual choice, including data revocation, against group consensus. The phases overlap and loop back on each other via two-way knowledge sharing.
Why is queer speech data considered especially sensitive?
Because common annotations like gender can reveal community membership ('outing') for non-binary and trans speakers; the community has a documented history of AI harms, including attempted 'gaydar' systems and queer dialects disproportionately flagged as toxic by moderation models; and qualitative studies report deep distrust of voice-AI developers among trans and non-binary users.
Has the framework been tested with a real community?
No. The authors present it as 'normative infrastructure' rather than a validated protocol, and explicitly defer application, iteration, and community feedback to future work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's schematics can be assessed only via their caption; the visual content is not analyzable from extracted text.
- Claims about third-party dataset composition (e.g., FairSpeech, Casual Conversations v2 speaker counts) could not be independently checked within the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.