Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models
Minji Kim, Jihyoung Jang, Hyounghun Kim
cs.CL · cs.AI
Quick verdict
KoNA shows VLMs fail selective non-compliance on compound queries; fine-tuning on 1.3K examples helps, but LLM-judge circularity and weak stats temper claims.
Summary
The paper introduces KoNA, a benchmark measuring whether vision-language models can withhold compliance selectively: answering valid components of a compound query while correcting, refusing, or abstaining on invalid ones. KoNA spans five categories (False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, Safety), each instantiated as paired single and compound queries plus a fully answerable contrast query, totaling 3,100 image-level instances and 9,300 QA pairs built from MS COCO and Open Images V7 via GPT-5 and Gemini-2.5-Flash generation, with automatic filtering and MTurk verification of all test instances. Evaluating InternVL3-2B/78B, Qwen2.5-VL-3B/72B, GPT-5, and Gemini-2.5-Flash, the authors find compound-query accuracy consistently below single-query accuracy (InternVL3-2B falls from 0.25 to 0.10 overall under default inference). Fine-tuning two 3B models with SFT plus GRPO on 1,300 training examples sharply raises non-compliance accuracy (0.92/0.82 single/compound for InternVL3-2B-KoNA) while keeping answerable accuracy near baseline, with CC3M out-of-distribution images, four extended external benchmarks (HaloQuest, MM-SafetyBench, R-Bench, UPD), and VizWiz human-originated queries supporting transfer. Main limitations: GPT-5-mini serves simultaneously as generator-side filter, judge for all metrics, and GRPO reward model, raising circularity concerns; no significance tests or confidence intervals are reported; and only 3B-scale models are fine-tuned.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing benchmarks evaluate non-compliance only at the whole-query level, whereas real requests mix answerable and non-compliant components; KoNA operationalizes component-level selective non-compliance via paired single and compound queries across five categories.
- Current VLMs, open and closed source, show compound-query accuracy consistently below single-query accuracy (e.g., InternVL3-2B: 0.25 vs 0.10 overall under default inference).
- SFT followed by GRPO on about 1,300 KoNA examples raises non-compliance accuracy sharply (InternVL3-2B-KoNA: 0.92/0.82 single/compound) while largely preserving answerable-task and general-capability performance.
- The fully answerable contrast set is necessary to prevent over-refusal (answerable average 0.38 without it vs 0.60 with it for InternVL3-2B), and GRPO further improves the compliance balance (to 0.70).
- The learned behavior transfers to CC3M out-of-distribution images, four extended external benchmarks, and a filtered set of VizWiz human-originated queries (0.32 to 0.72 and 0.39 to 0.73 selective non-compliance rates).
Strengths
Genuinely new evaluation axis with a controlled design
The paired single/compound query construction (Sec 3.2, Figure 2) isolates component-level selective non-compliance from query-level refusal, which prior benchmarks (HaloQuest, MM-SafetyBench, R-Bench, UPD) do not measure; the fully answerable contrast query additionally controls for over-refusal.
Unusually thorough judge validation
GPT-5-mini is checked against human judgments on 640 outputs (94.0-97.0% agreement by metric, 94.8% overall; App C.2), against Gemini-2.5-Flash (95.2% overall), and its GRPO reward against two alternative judges (85.2%/90.1% exact match, Pearson 0.887; App B.1), with a disagreement analysis suggesting the judge errs strict (App G).
Breadth of robustness analyses beyond the main table
Data-efficient training with a clear mechanism story
Dataset quality control
Weaknesses
Generator/filter/judge/reward role concentration in one model family
No inferential statistics anywhere
Train/test entanglement within one construction pipeline
Answerable-accuracy cost slightly understated
Fine-tuning evidence restricted to 3B scale
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
KoNA (arXiv 2609.04720, POSTECH) attacks a real gap: prior non-compliance benchmarks score whole-query refusal, while real user requests mix answerable content with components that should trigger correction, abstention, or refusal. The benchmark defines five categories — False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, Safety (Sec 3.1) — and for each image emits a single QA, a compound QA (invalid component plus answerable component), and a fully answerable contrast QA (Sec 3.2). Three claims are made: (1) VLMs degrade from single to compound queries; (2) SFT+GRPO on KoNA restores selective non-compliance while preserving answerable behavior; (3) the answerable set and GRPO each play distinct, necessary roles. All three are supported by the presented experiments, subject to the caveats below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is selective non-compliance in vision-language models?
It is the ability to answer the valid parts of a user request while refusing, correcting, or abstaining only on the invalid parts — for example, describing a visible object but declining an embedded unsafe action, or correcting a false premise instead of accepting it. The KoNA paper argues prior benchmarks only test all-or-nothing refusal per query.
What is the KoNA benchmark and how big is it?
KoNA evaluates selective non-compliance across five categories (False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, Safety). It contains 3,100 image-level instances and 9,300 QA pairs from MS COCO and Open Images V7 images, with paired single and compound queries plus fully answerable contrast queries; all test instances were human-verified via MTurk.
How well do current VLMs handle compound queries with an invalid component?
Poorly. Compound-query accuracy is consistently below single-query accuracy across tested models; under default inference InternVL3-2B averages 0.25 on single queries but only 0.10 on compound queries, and prompting strategies like chain-of-thought give only limited and inconsistent gains.
Does fine-tuning on KoNA hurt general model ability?
Mostly not, with small costs: TextVQA and POPE stay roughly unchanged, MIA-Bench and MMBench drop slightly (e.g., MMBench 0.80 to 0.76 for InternVL3-2B), over-refusal on safe queries stays low (MOSSBench 0.01 to 0.04), while non-compliance accuracy rises dramatically (0.92/0.82 single/compound for the tuned 2B model).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The text is a sampled excerpt: one sampling gap falls inside Sections 4.3-5, cutting Table 2 after the InternVL3-2B row (InternVL3-78B, Qwen, GPT-5, and Gemini rows are not visible), so cross-model single/compound numbers cited here come from Tables 3-6 and 11, which are fully shown.
- In Table 4 rows only five of the six single/compound column pairs are visible in the extraction (the Safety column pair is dropped); overall and answerable averages used in this review are unambiguous.
- A second sampling gap truncates part of the references and Appendix L prompt tables; all prompts and appendix procedures cited in this review (Tables 12-23, 27-29; Apps A-K) fall within the visible text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.