Counterfactual Reasoning for Robust Visual Question Answering
Truong-Binh Duong, Thanh-Ngan Tran, Ngoc-Thao Nguyen, Bac Le
cs.CV
Quick verdict
Staged counterfactual contrastive training for VQA reaches the best annotation-free VQA-CP v2 accuracy (61.64%); gains are modest, single-run, and shown only on the 2018 UpDn backbone.
Summary
The paper proposes a training framework for robust Visual Question Answering that counteracts language bias, where models exploit question-answer priors and fail under the distribution shift of VQA-CP v2. Built on the UpDn backbone with CSS counterfactual sample synthesis, the framework adds three components: a SimCLR-style Batch-Contrastive loss using all in-batch negatives (replacing CL-VQA's single-pair objective), an Answer-Contrastive loss aligning features with GloVe embeddings of correct answers, and a Gradient-Discrepancy loss penalizing similar sensitivities to factual versus counterfactual views, integrated via a three-stage curriculum over 30 epochs. The model reports 61.64% on VQA-CP v2 (highest among annotation-free methods listed) and 62.80% on VQA v2, a 1.16-point gap. Ablations show joint training without the curriculum collapses to 52.94%, BC beats the original contrastive loss by 0.82 points, and AC/GD help mainly in combination with BC, with GD boosting Yes/No to 90.52% but hurting Number on the CSS-only baseline. Limitations: evaluation is confined to the 2018 UpDn backbone, results lack variance estimates or significance testing, the AC loss is cited from prior work despite being labeled novel, Number accuracy trails KDDAug by over four points, and the GD loss equation is under-specified.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A three-stage curriculum is necessary for stable joint optimization: training all losses from scratch collapses VQA-CP v2 accuracy to 52.94% versus 61.64% with staging (Table 5).
- The SimCLR-style Batch-Contrastive loss outperforms CL-VQA's original contrastive loss by 0.82 points (61.64% vs 60.82%) and unlocks synergy with the AC and GD regularizers (Tables 2 and 4).
- The framework achieves the highest overall VQA-CP v2 accuracy (61.64%) and best Yes/No score (90.44%) among the annotation-free methods compared (Table 1).
- Answer-distribution analysis shows the model adapts to test-set priors (e.g., 'green' for bananas, 'yes' for 'Is this...?'), providing direct evidence of reduced reliance on language bias (Section 4.4).
Strengths
Unusually candid ablation reporting
Section 4.5 / Table 2 openly reports negative results: GD alone on the CSS baseline drops Number accuracy from 49.42 to 45.17, and AC+GD adds nothing over GD alone; the authors frame the regularizers as requiring BC's feature space rather than hiding this.
Curriculum necessity is convincingly demonstrated
Table 5 shows joint training from scratch collapses to 52.94%, two-stage reaches 61.39%, and the proposed three-stage schedule reaches 61.64% on VQA-CP v2 — a clean, decisive comparison.
Comprehensive, annotation-aware baseline comparison
Behavioral evidence beyond aggregate accuracy
Strong reproducibility detail in the text
Weaknesses
Novelty overstatement for the AC loss
No uncertainty quantification
Outdated backbone, untested transfer
Number questions remain weak
GD loss under-specified and cost unreported
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What accuracy does this paper achieve on VQA-CP v2?
The model reports 61.64% overall accuracy on VQA-CP v2 (90.44% Yes/No, 51.74% Number, 49.26% Other), described as the best among annotation-free methods in its comparison table, while keeping 62.80% on standard VQA v2 — a generalization gap of 1.16 percentage points between the two benchmarks.
What is the Gradient-Discrepancy (GD) loss in this paper?
It is a regularizer that penalizes the model when its sensitivity (output gradients with respect to visual region features) is similar for factual and counterfactual samples. A large discrepancy implies the model attends to the causal image regions that were masked in the counterfactual view, rather than answering from question-answer statistical shortcuts; it is weighted by the contrastive pair probability so the signal focuses on uncertain cases.
How is this method different from CL-VQA?
It builds on CL-VQA's idea of contrasting original, factual (CSS-derived positive), and counterfactual samples, but replaces the single positive/negative pair objective with a SimCLR-style batch-contrastive loss that uses all in-batch negatives, adds two regularizers (Answer-Contrastive loss on GloVe answer embeddings, and Gradient-Discrepancy loss for visual grounding), and introduces them gradually via a three-stage curriculum over 30 epochs.
Is the code for this paper available?
The paper states that code, model configurations, hyperparameter settings, and evaluation instructions are available at a GitHub repository; however, the concrete URL is not visible in the extracted text, so availability should be verified before relying on it.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 are not analyzable from the extracted text; judgments about qualitative results and the GD-loss illustration rely on captions and the prose of Sections 4.3-4.4.
- Equations 1-3 are garbled by PDF extraction; Eq. 3 may have lost an absolute-value operator, so part of the GD-loss critique may reflect extraction artifacts rather than the manuscript.
- The GitHub repository URL is absent from the extracted text; code availability is claimed but could not be verified.
- Table values were reconstructed from linearized text; minor misalignment between cells is possible though headline numbers are internally consistent across sections.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.