I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Models
Aydin Javadov, Daniel Schoess, Florian von Wangenheim
cs.LG · cs.CL · cs.CV
Quick verdict
VLMs overstate modality sufficiency: 62/64 cells underpredict restoration effects; rigorous interventional protocol, but only 2 families, non-thinking mode.
Summary
The paper asks whether vision-language models can faithfully report how missing input modalities affect their own predictions. The authors introduce an interventional protocol along two axes: retrospective attribution (does the stated single-modality account match executed single-input runs?) and prospective counterfactual self-knowledge (does the predicted answer after restoring a withheld modality match the executed restoration?), with claims graded in the same answer space as behavior under greedy decoding. Eight open-weight models (Qwen3.5 4B-35B and Gemma 4 2B-31B) are tested on four tasks: mm-IMDb with complementary modalities, two isomorphic IsoBench tasks, and GoldenView multi-view driving QA with annotated evidence. Models massively underpredict restoration effects: median predicted change never exceeds 8.8% while executed change reaches 72.1% on mm-IMDb, with underprediction in 62 of 64 model-task-condition cells. Insufficiency flags are precise (median 78-100%) but rare (recall 12.1% on mm-IMDb), leaving most answers that would change as 'silent sufficiency failures'. Retrospective attributions over-credit single-modality sufficiency on complementary data and interchangeability on isomorphic data. On GoldenView, stated attributions localize the annotated evidence better than answer reproduction does. Limitations: only two model families in non-thinking mode, a 300-film mm-IMDb subsample, only 55 GoldenView questions, and a self-report prompt costing a median 3.6pp task performance and changing many answers (mitigated by matched-template checks).
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- VLMs systematically underpredict the behavioral impact of restoring a missing modality: median predicted change rates never exceed 8.8% while executed change rates reach 72.1% (mm-IMDb), with underprediction in 62 of 64 model-task-condition cells.
- Insufficiency claims are precise when produced (median 78-100% of flagged cases would indeed change) but severely under-produced (recall 12.1% on mm-IMDb), so most changed answers sit under a SUFFICIENT claim as 'silent sufficiency failures'.
- Retrospective attributions over-credit sufficiency: on complementary data models claim single-modality sufficiency more often than executed (mm-IMDb stated TEXT 47.5% vs executed 26.3%), and on isomorphic data they claim interchangeability more often than behavior supports (EITHER claimed 92.1%/86.9% vs executed 79.2%/55.1%).
- On GoldenView, stated attributions localize the benchmark-annotated evidence set better than answer reproduction does, while the annotation itself has behavioral force (median 31.7pp accuracy gap for the golden set).
Strengths
Executable faithfulness criterion with matched claim/outcome spaces
Section 3 grades every stated claim against behavior on the same instance, in the same answer space, via the same attribution rule δ (Eq. 1-2); greedy decoding with one generation per arm removes decoder noise, and prompts never name an input the model was not shown (except the fixed-rig GoldenView case, footnoted).
Confound control is unusually rigorous
Appendix A.3 shows the assessment template alone changes parsed answers on a median 59.8% of mm-IMDb inputs, then re-derives the headline gap under two matched wirings (Table 8): executed change stays at 60.6-71.0% vs 8.8% predicted on mm-IMDb, and the underprediction count holds at 60-63 of 64 cells; Table 9 shows retrospective gaps widen (executed BOTH 41.9%→54.1%) under template matching.
Prompt design anticipates known response biases
Evidence-grounded validation on GoldenView
Transparent attrition and noise accounting
Weaknesses
Narrow model coverage limits the generalization
Thin data at the task edges
The measured model is template-mediated
Chance agreement floors are discussed but never corrected
Purely diagnostic; coarsest intervention only
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses a narrow but well-posed question: can VLMs faithfully self-report how missing input modalities affect their predictions? Two claim families. Prospective: models underpredict the effect of restoring a withheld modality (Table 3: predicted-change medians at most 8.8% vs executed up to 72.1% on mm-IMDb; underprediction in 62/64 model-task-condition cells), and insufficiency flags are precise but rare (Table 4: precision 78-100% median, recall 12.1% on mm-IMDb). Retrospective: stated attributions diverge from executed ones (Table 2: mm-IMDb stated TEXT 47.5% vs executed 26.3%, BOTH 14.8% vs 41.8%; IsoBench EITHER claimed 92.1%/86.9% vs executed 79.2%/55.1%). The faithfulness criterion is explicitly behavioral and output-level (Section 3, footnote 2, citing Parcalabescu & Frank [21]); no mechanistic claim is made.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
Vision-language models misreport how missing modalities affect them: they overwhelmingly claim the available input is sufficient, yet restoring the missing modality changes their answers far more often than predicted (median predicted change at most 8.8% vs executed change up to 72.1%; underprediction in 62 of 64 model-task-condition cells).
Which models and tasks were evaluated?
Eight open-weight VLMs from two families — Qwen3.5 (4B, 9B, 27B, 35B-A3B) and Gemma 4 (E2B, E4B, 26B-A4B, 31B) — in non-thinking mode, on mm-IMDb genre classification, two isomorphic IsoBench tasks (function parity, graph max-flow), and GoldenView multi-view driving QA.
What is a 'silent sufficiency failure'?
A case where the model says its available evidence is sufficient, but supplying the missing modality would actually change its answer — the model acts on incomplete information and signals nothing. On mm-IMDb, 88% of answers that the intervention would change sat under a SUFFICIENT claim.
Does the paper propose a fix for the problem?
No — it is diagnostic. It motivates executable modality interventions as behavioral ground truth for evaluating multimodal self-explanations, but tests no mitigation such as calibration training or prompt changes.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted bibliography ends at entry [27]; entries [28]-[30] (nuScenes, Matton et al., vLLM) are cited in appendices B.3/C.2/E.2 but their reference-list entries are absent from the extraction.
- Figures 1-5 appear only as extracted caption and axis-label text; plot content could not be visually verified, so figure-level readings rely on the captions and surrounding prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.