Benchmarking the Explanatory Quality of Open-Weight Vision-Language Models in Face Recognition
Laurent Colbois, Sébastien Marcel
cs.CV · cs.AI
Quick verdict
Introduces relevance/faithfulness metrics for VLM face-recognition explanations via structured outputs; a sound benchmark, though lexicon metrics stay coarse.
Summary
The paper introduces a benchmarking framework that evaluates open-weight vision-language models (VLMs) for face verification not only on recognition accuracy but also on the quality of their natural-language explanations, targeting forensic use cases. The authors constrain model outputs to a structured JSON schema via constrained decoding, enabling automated auditing of explanations. They propose two dataset-driven metrics: relevance, the share of feature comparisons that avoid forensically unstable cues (glasses, expression, lighting, attire) via a keyword lexicon; and faithfulness, the avoidance of describing facial regions known to be occluded, using occlusion labels from ARFace and Soteria. Benchmarking Gemma3 (4B–27B), Qwen2.5-VL (3B–72B), and InternVL3 (1B–78B) on LFW, ARFace, Soteria, and CelebA protocols, they find VLMs remain well behind ArcFace (best VLM EER 5.2% on LFW vs 0.3%), structured decoding imposes an EER cost that shrinks with scale, and explanation quality varies independently of accuracy—relevance often peaks at intermediate model sizes and degrades for the largest models, while faithfulness improves more consistently with scale. Lexicon-based relevance scoring is admittedly conservative, faithfulness is restricted to occlusion, and possible pretraining overlap with LFW/CelebA may confound results.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Explanation quality (relevance and faithfulness) varies substantially across VLMs independently of verification accuracy, so accuracy-only benchmarks give an incomplete picture.
- Enforcing structured JSON explanations generally costs EER, and the cost shrinks with model scale.
- Relevance does not improve monotonically with scale: intermediate-sized models (Gemma3-12B, Qwen2.5-VL-7B, InternVL3-14B) score best and the largest models often degrade.
- Faithfulness to occlusions tends to improve more consistently with scale.
- VLM face verification remains far behind dedicated FRSs (best VLM EER 5.2% on LFW vs 0.3% for ArcFace Buffalo-L).
Strengths
First quantified explanation-quality axis for VLM face recognition
§2.3 shows prior benchmarks discuss explanations only qualitatively; §3.2.2 operationalizes relevance (unstable-cue lexicon) and faithfulness (occlusion labels) into automatable rates — a genuine gap-filler for the forensic audience.
Structured output avoids judge-model error
Constrained decoding into a JSON schema (Fig. 2) makes scoring a deterministic heuristic instead of LLM-as-judge — the exact failure mode the authors attribute to FaithScore [12] — while leaving the feature list free keeps models expressive yet queryable.
Broad, controlled model and protocol coverage
Honest operational-cost accounting
Concrete failure taxonomy from automated audit
Weaknesses
Lexicon metrics never validated against humans
No uncertainty quantification
Faithfulness is occlusion-only
Scoring surface is one field of the schema
Protocol and release ambiguities
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper argues that benchmarking VLM-based face verification should measure explanation quality, not just accuracy. It introduces two forensic-motivated criteria — relevance (reliance on identity-stable facial features rather than transient cues such as glasses, expression, or lighting) and faithfulness (no descriptions of facial parts invisible due to occlusion) — plus a methodology that quantifies both by constraining outputs to a structured JSON schema (Fig. 2). Central claims: (1) prior evaluations leave explanation validity unquantified (§2.3); (2) structured outputs enable automated auditing without LLM-as-judge; (3) across three open-weight families, explanation quality varies independently of EER, structured decoding carries an EER cost that shrinks with scale, and relevance is not monotonic in scale. Grounding in FISWG/ENFSI forensic standards (§2.1) is well motivated and rare in this literature.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do vision-language models reliably explain their face recognition decisions?
This paper measures exactly that. It scores explanations from open-weight VLMs on relevance (using stable facial features rather than transient cues like glasses, expression, or lighting) and faithfulness (not describing features hidden by occlusion). Finding: explanation quality varies widely across models and is unrelated to verification accuracy — models with nearly identical accuracy can differ sharply in how valid their explanations are.
How accurate are VLMs at face verification compared to dedicated face recognition models?
Still far behind. In this benchmark the best VLM configurations (Gemma3-27B and Qwen2.5-VL-72B) reach about 5.2% EER on LFW, versus 0.3% for a state-of-the-art ArcFace model (Buffalo-L). Small VLMs (1B–3B) are near chance level, and the smallest often fail to produce valid structured output at all (up to ~29% failure rate).
What are the relevance and faithfulness metrics in this benchmark?
Relevance is the share of feature comparisons that avoid forensically unstable cues (glasses, expression, lighting, attire, grooming), counted by a keyword lexicon. Faithfulness is 1 minus the rate of mentioning facial parts that are occluded and thus invisible, using occlusion labels from the ARFace and Soteria datasets; mentions that merely acknowledge the occlusion are not penalized.
Is the benchmark code available?
The paper points to a public GitLab repository (gitlab.idiap.ch/biometric/vlmfr) for the benchmarking framework and states the code is released publicly; a footnote also says the URL for prompts, JSON schemas, and evaluation code will be provided upon acceptance, so completeness should be verified directly in the repo.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1, 3 and 4 are only partially recoverable from extracted text; Fig. 3 scaling claims were cross-checked against the numbers quoted in §4.3 prose.
- Dense multi-column tables (Tables 2–3) risk cell misalignment in extraction; headline values were verified against prose mentions where possible.
- Supplementary material is only partially available (Table 5 reproduces the full lexicons; nothing beyond that was assessable).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.