FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making
Vahid Reza Khazaie, Ahmed Y. Radwan, Shaina Raza
cs.CV · cs.LG
Quick verdict
A VLM fairness benchmark making abstention a first-class metric across 8 models; valuable design, yet no CIs/significance tests and an unvalidated LLM judge.
Summary
FairLens is a benchmark and evaluation framework probing whether vision-language models make unsupported, demographically patterned judgments from face images in hiring, legal, and healthcare settings. The authors pair 1,505 stratified UTKFace adult images (ages 25–65, with gender, race, and age annotations) with 69 fixed questions (38 multiple-choice, 31 open-ended), yielding 103,845 image-question pairs per model across eight VLMs (CogVLM, GPT-5.2-Reasoning, InternVL3, LLaMA-3.2-Vision, LLaVA-1.6, Ovis2.5, Qwen2.5-VL, Qwen3-VL). Four complementary metrics are computed: demographic parity gaps over adverse outcome rates, soundness (following context-stated evidence or abstaining with 'Cannot tell' when the image cannot support an answer), demographic association rates over unsupported roles/statuses, and LLM-judge bias in free text. Results: Ovis2.5 leads soundness (93.8%); LLaMA-3.2-Vision answers 99% of image-insufficient questions non-neutrally yet shows the smallest association gap (0.7pp), demonstrating that parity-style metrics miss uniformly unsafe behavior. Parity gaps are small (about 1–3pp) but rate ratios reach 9.0x (InternVL3, healthcare race) from baselines as low as 0.1%. Soundness and open-ended bias decouple (GPT-5.2: 77.1% soundness, 1.8% bias; InternVL3: 88.1%, 10.0%). Main limitations: no confidence intervals or significance tests, small demographic subgroups, an unnamed and unvalidated LLM judge, and coarse binary gender labels in UTKFace.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The dominant failure mode of VLMs on high-stakes face-based questions is unwarranted inference rather than unequal demographic treatment: the weakest model (LLaMA-3.2-Vision) gives non-neutral answers on 99% of image-insufficient questions.
- Small percentage-point demographic parity gaps can mask large adverse-outcome rate ratios at low baselines (e.g., a 0.8pp gap corresponding to a 9.0x ratio), so parity metrics alone are misleading measures of fairness.
- Structured-answer soundness and free-text bias are only loosely coupled: GPT-5.2-reasoning scores 77.1% soundness with the lowest bias rate (1.8%), while InternVL3 scores 88.1% soundness with 10.0% bias.
- Legal and healthcare questions are hardest for evidence recognition (average soundness 60.0% and 56.5% vs 82.2% in hiring), precisely the domains where abstention matters most.
- A model can treat all demographic groups similarly while being uniformly unsafe, so fair high-stakes behavior requires both parity and refusal to infer high-stakes attributes from appearance.
Strengths
Abstention elevated to a first-class validity metric
Soundness (Eqs. 1-2) scores whether models follow stated evidence or answer 'Cannot tell' when the image cannot support an answer. The LLaMA-3.2-Vision result (99.0% association rate with the smallest association gap, 0.7pp, Table 2) concretely demonstrates the decoupling that parity-only audits miss.
Careful disparity reporting
Table 5 reports percentage-point gaps alongside rate ratios (e.g., InternVL3 healthcare race 0.8pp but 9.0x; LLaMA-3.2-Vision 11.8pp but only 1.3x), and Section 4.4 explicitly declines to convert domain-average gaps into 'times higher' factors — an unusually honest treatment of the low-baseline problem.
Full stimulus transparency and reproducible closed-ended protocol
Breadth of evaluation
Real faces with neutral framing
Weaknesses
No uncertainty quantification anywhere
Headline rate ratios rest on tiny event counts
LLM judge unnamed and unvalidated
Soundness conflates parsing with substantive failure
Aggregation and confound opacity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FairLens (styled FAIRLENS in much of the text) is a benchmark and evaluation framework asking whether vision-language models make unsupported, demographically patterned judgments from face images in hiring, legal, and healthcare contexts. Eight VLMs are evaluated (CogVLM 19B, GPT-5.2-reasoning via API, InternVL3 9B, LLaMA-3.2-Vision 11B, LLaVA-1.6 7B, Ovis2.5 9B, Qwen2.5-VL 7B, Qwen3-VL 8B) over 1,505 stratified UTKFace adult images (ages 25-65) and 69 fixed questions (38 multiple-choice, 31 open-ended), i.e., 103,845 image-question pairs per model. Four claims anchor the paper: (i) the dominant failure is unwarranted inference rather than unequal treatment - the weakest model answers 99% of image-insufficient questions non-neutrally (Table 2); (ii) percentage-point parity gaps mislead at low baselines (Table 5); (iii) structured soundness and free-text bias are loosely coupled (Section 4.5); (iv) the question suite transfers to any demographically annotated face corpus (Section 5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the FairLens benchmark and what does it measure?
FairLens (arXiv 2609.01691, Vector Institute) is a benchmark and evaluation framework testing whether vision-language models make unsupported or demographically patterned judgments from face images in hiring, legal, and healthcare scenarios. It pairs 1,505 real UTKFace adult face images with 69 fixed questions per image and scores responses on four views: demographic parity of adverse outcomes, soundness (following stated evidence or abstaining with 'Cannot tell'), demographic association with unsupported roles and statuses, and LLM-judged bias in free text.
Which vision-language model performed best on FairLens?
No model wins on all metrics. Ovis2.5 has the highest soundness (93.8%) and the lowest association rate (5.4%); GPT-5.2-reasoning has the lowest parity gap (0.2pp) and the lowest open-ended bias rate (1.8%). LLaMA-3.2-Vision is weakest: 21.9% soundness and non-neutral answers on 99% of image-insufficient questions. The paper reports no statistical tests, so these rankings are descriptive rather than confirmed as significant.
Why does the paper argue demographic parity alone is misleading for fairness evaluation?
Because parity gaps are measured in percentage points, a small gap at a low baseline can still mean one group receives adverse labels several times as often as another: InternVL3's healthcare race gap is only 0.8pp but its Black adverse rate is 9.0x its Asian rate. Additionally, a model that behaves unsafely toward every group produces tiny gaps. FairLens therefore reports gaps alongside rate ratios and adds validity metrics such as soundness and abstention analysis.
What data does FairLens use and is the benchmark available?
It uses 1,505 real adult face images (ages 25-65) sampled with a fixed seed from the public UTKFace dataset, stratified by binary gender, five race categories, and three age bands, crossed with 69 questions (38 multiple-choice, 31 open-ended) for over 100,000 image-question pairs per model. The paper states that code and benchmark resources are available at the project's Vector Institute GitHub Pages site.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The paper-text extraction is essentially complete (through Appendix C, Tables 6-12); Figure 1 is available only as extracted text, so its visual layout was not assessed.
- The abstract in the task metadata is cut mid-sentence, but the complete abstract appears at the start of the extracted paper text, so nothing relevant was missing for this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.