The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence
cs.CV · cs.CL · cs.LG
Quick verdict
Proposes VSI, a blur-perturbation KL measure showing 40–97% of VLM samples ignore visual evidence; careful, honest study but diagnostic utility is narrow.
Summary
The paper introduces the Visual Sensitivity Index (VSI), a per-sample diagnostic measuring the KL divergence between a VLM's next-token distribution before and after Gaussian-blurring the question-relevant image region, located via GroundingDINO+SAM. Across six VLMs (LLaVA-1.5, LLaVA-NeXT, Idefics3, Qwen3-VL-8B, Qwen2.5-VL-7B/32B) and three perceptual benchmarks (POPE, MMVP, HallusionBench), 40–97% of samples fall below VSI<0.05. VSI ranks correlate across models (grand-mean Spearman ρ=+0.40; permutation p<10^-3), supporting a sample-intrinsic phenomenon. On low-VSI samples, linear probes on each model's own vision tower detect the perturbation (0.72–0.79 accuracy; frozen CLIP 0.82) while argmax tokens change on only 2–11% of the same samples — an encoder–LLM gap above 0.65 everywhere. Diagnostically, VSI-containing hybrids win 7 of 18 selective-generation cells and softmax wins 10; VSI shines mainly on MMStar math/science with Qwen2.5-VL-32B (AUROC 0.851/0.867, n=34/46, wide CIs) and inverts on MMVP. Limitations include modest cross-model correlations, KL computed on top-50 token support, a 19% region-detection fallback on HallusionBench, and heavy exclusion of degenerate decoding outputs for Qwen models (67% of one POPE cell).
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Blurring the question-relevant visual region leaves the next-token distribution nearly unchanged (VSI<0.05) on 40–97% of samples across six VLMs and three perceptual benchmarks.
- VSI ranks correlate across architecturally disjoint models (grand-mean Spearman ρ=+0.40 over 15 pairs, permutation p<10^-3), making visual insensitivity a sample-intrinsic rather than model-specific property.
- On low-VSI samples, each model's own vision tower linearly encodes the perturbation (0.72–0.79 accuracy) while its argmax token changes on only 2–11% of the same samples, an encoder–LLM gap above 0.65 on every model.
- VSI is a conditional, not universal, abstention signal: softmax wins 10/18 perceptual cells, VSI-containing hybrids win 7/18, and the strongest standalone regime is MMStar math/science on Qwen2.5-VL-32B (AUROC 0.851/0.867).
Strengths
Honest, conditional claim structure
§4.1 and Table 3 explicitly report softmax max-prob winning 10/18 cells and VSI standalone winning only 1/18; the abstract itself states VSI is 'not a universal best abstention signal'. The MMVP inversion (bottom-quintile error 15.0% vs top 25.0% on Qwen3-VL, §3.1) is reported in the main text, not buried.
Well-controlled mechanism experiment
§2.4/Table 2 pairs own-tower linear probes (0.722–0.789 on the shared 83-image low-VSI pool) with a matched high-VSI control (probe rises to 0.859–0.906, ruling out probe-capacity ceilings) and a frozen reference CLIP witness (0.82±0.04) independent of all six models (App. A.4).
Substantial robustness work
Unusual transparency on pathologies
Architecture diversity makes the cross-model claim testable
Weaknesses
Headline 'strong regime' rests on tiny cells
Probe may detect blur artifacts, not lost question-relevant content
VSI implementation diverges from its definition
Per-cell samples shift with parseability
No multiple-comparison control across the grid
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper defines the Visual Insensitivity Gap: the discrepancy between what a VLM's vision tower represents and what its language head emits. It operationalises this as the Visual Sensitivity Index (VSI, §2.1) — the KL divergence between next-token distributions for a clean image and the same image with the question-relevant region Gaussian-blurred (σ=20; region found via GroundingDINO+SAM, with <3% whole-image fallback on POPE/MMVP but ~19% on HallusionBench). Three claims are tested: (1) the gap is large — 40%–97% of samples have VSI<0.05 across six models × three perceptual benchmarks (Fig. 1); (2) it is sample-intrinsic — VSI ranks correlate across models (grand-mean Spearman ρ=+0.40 over 15 pairs, permutation p<10^-3, §2.3); (3) the mechanism is an encoder–LLM disconnect (§2.4), with VSI positioned as a conditional, not universal, abstention signal (§4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Visual Insensitivity Gap?
It is the discrepancy between what a vision-language model's vision encoder represents and what its language head outputs. On 40–97% of benchmark samples, blurring the question-relevant image region barely changes the model's answer distribution, even though a linear probe on the model's own vision encoder detects the change.
How is the Visual Sensitivity Index (VSI) computed?
The object a question asks about is located with GroundingDINO and refined to a mask with SAM, then Gaussian-blurred (σ=20). VSI is the KL divergence between the model's next-token distributions for the clean and blurred images; low VSI means the model's output distribution ignores the visual change.
Can VSI replace softmax confidence for deciding when a model should abstain?
No. The paper shows softmax max-probability is the best signal on 10 of 18 model–benchmark cells; ensembles containing VSI win 7 cells, mainly where calibration is poor or the failure mode is hallucination. VSI is recommended as a conditional ensemble component, not a replacement.
Which models and benchmarks were tested?
Six open VLMs — LLaVA-1.5-7B, LLaVA-NeXT-Mistral-7B, Idefics3-8B, Qwen3-VL-8B, Qwen2.5-VL-7B and Qwen2.5-VL-32B — on POPE, MMVP and HallusionBench (perceptual) plus MMStar (multi-choice reasoning).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded numbers are partially garbled by PDF extraction (e.g., Figure 1 axis labels interleaved with sample counts); figure-only values were cited only when restated in captions or body text.
- The extracted text states code and per-sample CSVs are 'released' but contains no repository URL, so release completeness could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.