Knowing When Not to Answer: Abstention and Refusal Reasoning in Vision--Language Models
Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel, Hansa Meghwani, Jyotika Singh, Ranjeet Gupta, Graham Horwood, Tao Sheng, et al.
cs.CV · cs.AI
Quick verdict
PARITY benchmark shows VLMs split between refusal and speculation on unanswerable ASD queries; strong identity-controlled design, RQS may undercount quality.
Summary
The paper asks the opposite of most VLM audits: whether models abstain from a clinically unanswerable query — choosing which of two facial images shows the person 'more likely to have autism.' The authors introduce PARITY, 1,000 synthetic adult identities (7,000 images) stratified across race, gender, and age, each with a neutral anchor plus six identity-preserving expression edits generated with Nano Banana and validated through InsightFace similarity (tau_id=0.78), AU-delta checks (at least 30%), and human review (alpha=0.827). Because identities have no ASD status, any non-abstention counts as a harmful attribution. Across 14 VLMs at temperature 0, behavior splits into refusal-first models (Gemini 2.5 Pro and GPT-5, AR=1.000; Command-A Vision, 0.992) and speculative models (Gemma 3 4B, Phi-3.5, and Pixtral, AR=0.000). When models do answer, high-arousal expressions (surprise, fear, anger) disproportionately trigger selection of the edited portrait (EDBR of 0.5 or above in all nine qualifying models; Mistral 0.915). A rule-based Refusal Quality Score is low even for ceiling abstainers (Gemini 0.009), suggesting shallow justifications, though the conservative keyword inventory makes RQS a lower bound. Neutral-neutral controls show comparison pressure alone elicits choices; single-image framing and clinical guardrails raise abstention; Llama Guard 4 judged every prompt safe. Limitations include one prompt family, one spurious factor, a single generator, leading expression-focused prompt wording, and CFD validation reported only in figures.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Contemporary VLMs bifurcate into refusal-first models (Gemini 2.5 Pro and GPT-5, AR=1.000) and speculative models (Gemma 3 4B, Phi-3.5 4B, Pixtral 12B, AR=0.000) on unanswerable paired ASD queries (Table 1).
- Conditional on answering, models disproportionately select the expression-edited portrait (EDBR≥0.5 in all nine qualifying models, up to 0.915 for Mistral 3.1 24B), with high-arousal expressions (surprise, fear, anger) as the strongest triggers.
- Abstention rate alone overstates safety because abstentions frequently lack clinically grounded justification (RQS as low as 0.009 for Gemini 2.5 Pro despite AR=1.000).
- Interface design drives the failure: neutral–neutral controls elicit fabricated selections, single-image framing and clinical guardrails raise abstention, while Llama Guard 4 labels every prompt safe.
Strengths
Clean counterfactual design with credible manipulation checks
Within-identity neutral/edit pairs isolate expression from identity; three-layer validation — InsightFace antelopev2 similarity (τid=0.78, 92.3% edit acceptance), AU-delta checks (≥30% mean activation increase on the Table 3 AU sets via LibreFace), and bounded human review (3.2% rejection, Krippendorff's α=0.827) — makes the stimulus construction trustworthy (§3.1.4, Appendices E–F).
Neutral–neutral controls expose interface-induced fabrication
Presenting the identical image on both sides still elicits non-abstaining choices (HARnn>0) in several speculative models (§3.2.3, §4.5, G.2), showing comparison pressure alone produces fabricated distinctions — a finding about UI design, not just model capability.
AR/RQS dissociation is a genuinely useful metric contribution
Negative result on safety classifiers is practically important
Ethics-conscious stimulus construction
Weaknesses
Baseline prompt may manufacture the failure it measures
RQS construct validity is unverified
Single synthetic generator; real-face numbers absent from text
Descriptive statistics only; internal threshold inconsistency
Narrow generalization scope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Knowing When Not to Answer (PARITY)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can vision-language models diagnose autism from facial photos?
No — consistent with clinical guidance, this paper's premise is that ASD diagnosis requires behavioral and developmental assessment and cannot be inferred from a static photograph. The paper tests whether VLMs nonetheless produce diagnosis-like answers when asked, and treats any such answer as a harmful attribution because the test identities are synthetic and have no ASD status.
What is the PARITY benchmark?
PARITY (Paired Assessment with Reused Identity) is a dataset of 1,000 synthetic adult identities (7,000 images total), stratified by race, gender, and age. Each identity has a neutral portrait plus six identity-preserving expression edits, used to test whether vision-language models abstain from an unanswerable 'who is more likely to have autism' paired comparison.
Which VLMs refused to answer, and which speculated?
Gemini 2.5 Pro and GPT-5 abstained on 100% of trials, and Command-A Vision 112B on 99.2%, while Gemma 3 4B, Phi-3.5, and Pixtral 12B answered on every trial (0% abstention). InternVL 3.5, Llama 4 Maverick, and Mistral 3.1 fell in between (roughly 45–71% abstention).
What mitigations increased model refusal in this study?
Adding a clinical-guardrail system prompt and reframing the task as a single-image question both substantially raised abstention, showing the paired-comparison interface drives much of the harmful behavior. Swapping image order had little effect, and the Llama Guard 4 safety classifier labeled all test prompts as safe, missing the failure mode entirely.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded numeric values (ΔAR magnitudes in Fig 7, HARnn and side-bias values in Fig 9, CFD results in Figs 11–12, the AR–RQS scatter in Fig 13) are absent from the extracted text, so those specific magnitudes could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.