Face-voice Association across LAnguages and Gender (FLAG) 2027 Challenge Evaluation Plan
Marta Moscati, Swapnil Khandoker, Muhammad Saad Saeed, Shah Nawaz, Fatima Noor, Rohan Kumar Das, Mubashir Noman, Junaid Mir, et al.
cs.CV
Quick verdict
FLAG 2027 adds Bengali and same-gender tests to face-voice association; protocol detailed but evidence is one single-run baseline, no significance analysis.
Summary
This paper presents the evaluation plan for the FLAG 2027 Challenge, which studies face–voice association under two stress conditions: cross-lingual speech and gender-homogeneous distractors. Face–voice association is cast as cross-modal verification: given a voice, retrieve the speaker's face from a gallery containing negatives. Building on the FAME 2024/2026 challenges, the authors extend MAV-Celeb with a new 100-speaker English–Bengali split (57 male, 43 female; 557 videos; ~71.5 hours; ~55k utterances) with language and gender annotations, evaluated under an unseen-unheard split. Two settings test gender reliance: standard and gender-constrained (same-gender negatives), each crossed with heard (English) and unheard (Bengali) languages; models are ranked by the mean of four EERs. A FOP baseline trained on English only shows higher EERs on Bengali (38.12 vs 32.54, dev standard) and under gender constraints (44.01, dev Bengali), supporting the claim that current models exploit language and gender shortcuts. The plan specifies submission formats, CodaBench hosting, submission caps, disqualification rules, and prizes. Limitations: a single baseline in one training configuration, single-run numbers without variance or significance testing, unspecified gallery/pair composition, and an unheard-language contrast confounded with the English-only voice encoder.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing face-voice association models degrade when evaluated on an unheard language (Bengali) relative to a heard language (English).
- Performance deteriorates in gender-constrained settings where all gallery negatives share the speaker's gender, indicating reliance on gender cues rather than identity-specific voice characteristics.
- A new 100-speaker English-Bengali extension of MAV-Celeb with language and gender annotations, under an unseen-unheard protocol, enables systematic study of both effects.
- EER averaged over the four protocol cells is a threshold-free ranking metric appropriate for the challenge, consistent with prior FAME 2024/2026 evaluation.
Strengths
Targets two genuine evaluation gaps in face-voice association
Sec. 1-3 motivate language shift and demographic-shortcut reliance, and the gender-constrained track (same-gender negative galleries) is a direct probe of gender shortcuts rather than a generic robustness test.
Concrete new resource with full per-language statistics
Table 1 documents the new 100-speaker English-Bengali split (57/43 gender split, 557 videos, 71.5 h, 55,164 utterances, ~4.6 s mean utterance length), enabling dataset-level comparison with MAV-Celeb and the FAME 2024/2026 lineage.
Operationally complete challenge specification
Continuity and support infrastructure
Baseline results are consistent with the motivating hypothesis
Weaknesses
All degradation claims rest on a single baseline in one configuration
Heard/unheard contrast is confounded
Verification protocol under-specified
Possible identity leakage via face pretraining is unaddressed
Naming and label inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the FLAG 2027 Challenge?
An IEEE Signal Processing Society-supported grand challenge that evaluates face-voice association models as a cross-modal verification task — given a voice, identify the speaker's face among distractor faces — under two stress conditions: an unheard language (Bengali) and galleries where all distractors share the speaker's gender. Entries are ranked by the average equal error rate over four test cells, hosted on CodaBench, with results due in late 2026.
What dataset does the FLAG 2027 Challenge use?
An extension of the MAV-Celeb dataset with a new split of 100 English-Bengali bilingual speakers (57 male, 43 female), totaling 557 videos, about 71.5 hours of audio-visual material, and roughly 55,000 utterances, annotated with language and gender. Splits follow an unseen-unheard configuration: identities used for training never appear in development or evaluation, and each training speaker speaks only the training language.
How are submissions ranked, and what did the baseline show?
Each submission produces confidence scores for four protocol cells (heard English vs. unheard Bengali, crossed with standard vs. same-gender gallery), and the ranking metric is the mean of the four equal error rates. The FOP baseline scored 32.54 EER on English vs. 38.12 on Bengali (dev, standard setting) and up to 44.01 in the same-gender Bengali condition, showing that performance degrades under language shift and gender constraints.
Who can participate and what are the rules?
Teams register via a Google form between 15 September and 15 October 2026. Pretrained face or voice encoders are allowed, but every team must submit a link to working code and a 2-page system description in the ICASSP template or be disqualified. Up to 150 submissions (maximum 15 per day) are allowed in the progress phase and 15 in the evaluation phase; the top five teams share a $5,000 prize pool.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 cannot be inspected; their content is inferred from captions only.
- The extracted abstract duplicates and truncates (e.g., 'distinguis'), and Table 2's caption says 'MAV-Celeb V4' while the body says 'extended MAV-Celeb' — a naming inconsistency in the source.
- Gallery sizes, negative counts, and per-cell pair numbers are absent from the text, limiting protocol-level judgments.
- Dataset, pre-extracted features, and the pretrained baseline are promised for release at challenge start, not independently verified as available at review time.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.