Vision And Text Transformer For Predicting Answerability On Visual Question Answering
Tung Le, Huy Tien Nguyen, Le Minh Nguyen
cs.CV · cs.AI
Quick verdict
Reframes VizWiz answerability as regression with a ViT+BERT model (AP 76.96); promising, but weak baselines and unproven novelty weaken conviction.
Summary
This preprint addresses the VizWiz answerability task: predicting, for an image-question pair from a blind user, whether a VQA system can produce a meaningful answer. The authors argue that prior practice - binary-mapping a VQA classifier output to an answerability score - discards information, and reframe the task as regression to a continuous score trained with MSE on 0/1 targets, matching the average-precision (AP) evaluation. VT-Transformer encodes questions with pre-trained BERT ([CLS] representation) and images with pre-trained ViT-B/16, projects both to a shared dimension, fuses them via element-wise multiplication or concatenation, and applies an FC + sigmoid head. On VizWiz 2020 (20,523 train / 4,319 val / 8,000 test, ~27-32% unanswerable) they report AP 76.96 and F1 67.26, versus 26.84/42.32 for VWTest (the unpublished 2020 challenge winner) and 52.22/41.85 for a self-reproduced, regression-adapted BERT-RG. Ablations favor ViT over ResNet-152/VGG-16 (~+10 AP with multiplication fusion), multiplication over concatenation (76.96 vs 74.91 AP), and pre-training (58.95 to 76.96 AP). Limitations: the architecture is a standard ViT+BERT fusion; baselines are weak, unpublished, or self-reproduced with a strikingly low winner AP; there is no significance testing, seed variance, threshold protocol for F1, or released code; and the claimed first-ever regression formulation is not substantiated against prior unanswerable-VQA literature.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Formulating VizWiz answerability as continuous-score regression better matches the AP evaluation metric than binary mapping from VQA classifiers (Section 1).
- VT-Transformer (pre-trained ViT-B/16 + BERT, multiply fusion, MSE-trained sigmoid head) achieves AP 76.96 and F1 67.26 on VizWiz 2020, outperforming VWTest (26.84/42.32) and BERT-RG-Regression (52.22/41.85) (Table 3).
- Vision Transformer image embedding outperforms ResNet-152 and VGG-16 by roughly 10 AP under multiplication fusion, and multiplication beats concatenation (Table 4).
- Pre-trained initialization is decisive for performance, raising AP from 58.95 to 76.96 (Table 5).
Strengths
Task framing coherently aligned with the evaluation metric
Reframing answerability as continuous-score regression is explicitly motivated by the AP metric's threshold sweep (Section 1, Eq. 4), a rationale that binary-mapping approaches lack; the compute-saving use case (filtering unanswerable queries before running a VQA model) is well argued.
Substantial ablation coverage
Tables 4-6 ablate the image encoder (ViT 76.96 AP vs ResNet-152 63.13 and VGG-16 70.75 with MUL fusion), the fusion operation (MUL 76.96 vs CAT 74.91 AP), pre-trained initialization (58.95 to 76.96 AP), and model size (B-16 vs smaller VT-16).
Practical, accessibility-motivated dataset with clear statistics
Key implementation choices disclosed
Weaknesses
Weak or unverifiable baselines
Limited architectural novelty
No statistical rigor
F1 computation protocol undefined
Novelty claims unsubstantiated by related work
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the VT-Transformer model for VQA answerability?
VT-Transformer is a dual-encoder model that scores whether a visual question can be answered: pre-trained BERT encodes the question (its [CLS] vector), pre-trained Vision Transformer B/16 encodes the image patches, both feature vectors are projected to a common size, combined by element-wise multiplication (or concatenation), and a fully-connected layer with sigmoid outputs a 0-1 answerability score trained with mean squared error.
What results does VT-Transformer report on VizWiz 2020?
The paper reports an average precision of 76.96 and an F1-score of 67.26 on the VizWiz 2020 answerability task, versus 26.84/42.32 for VWTest (the unpublished 2020 challenge winning system) and 52.22/41.85 for a regression-adapted BERT-RG baseline reproduced by the authors. These figures come from the paper itself and are not independently verified.
What is the VizWiz answerability task?
VizWiz contains images and spoken questions recorded by blind users. The answerability task predicts, before running a full visual question answering model, whether a given image-question pair can be answered at all; about 27-32% of samples are unanswerable. The official metric is average precision, computed on a hidden 8,000-sample test set through the EvalAI evaluation server.
Why do the authors formulate answerability as regression instead of classification?
They argue that mapping a VQA classifier's output to a binary answerable/unanswerable label discards information, whereas a continuous score aligns with the average-precision metric, which evaluates ranking quality across many thresholds. Their model is trained with mean squared error against 0/1 targets but emits continuous scores; the paper does not analyze calibration or threshold behavior of these scores.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are garbled in the extracted text, so the architectural diagrams and example images could not be assessed.
- External facts (the VWTest leaderboard value, the absence of prior publications on this task) could not be verified against outside sources from the text alone.
- Equation rendering is partially mangled in extraction (e.g., Eq. 2), leaving minor notational uncertainty; section and table content otherwise appears complete through the references.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.