What Looks Like a Capability Limit in Vision-Language Models Is a Readout Limit
cs.CV · cs.AI · cs.CL
Quick verdict
Argues VLM benchmark answer conventions, not capability, drive large score drops (48.5 pts) and rank inversions; a rigorous, pre-registered empirical study.
Summary
The paper argues that low scores on vision-language model benchmarks can be caused by how answer options are written rather than by model capability. Using a 'convention swap' design that varies only the surface description of fixed answer choices on identical items, the author tests color (CIELAB hue angles vs calibrated words) and location (English cell names vs pixel/normalized coordinates) tasks on synthetic stimuli and 200 COCO photographs. Qwen3-VL-4B drops from 68.5% to 20.0% (chance 11.1%) when options are given as absolute pixels, a paired gap of 48.5 points (exact McNemar, p=1.2e-21), while the same coordinate as question context costs a non-significant 3.5 points. Conventions also flip rankings: Qwen2.5-VL-7B and Qwen3-VL-4B tie under English names but differ by +39 points under pixels and -54.5 under normalized coordinates. A mispaired-name 'readership' test shows hue angles are followed below chance while normalized coordinates are followed at about 4x chance, though a pre-registered prediction that readership predicts accuracy failed. Five models also partition one color wheel into five vocabularies. Strengths include pre-registration, per-model calibration, and a candid audit of five scoring errors; weaknesses include one dataset, only two frontier models, different scoring instruments for API models, and code not yet publicly released.
Review complete
Assessment at a glance
3
risk flags identified
● high 0
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On identical COCO items, changing only the description of the nine answer options from English cell names to absolute pixel coordinates drops Qwen3-VL-4B from 68.5% to 20.0% (chance 11.1%), a 48.5-point paired gap.
- The convention cost is answer-side: the same pixel coordinate supplied in the question costs only 3.5 points and is not significant, so the model is not generally failing at coordinates.
- Answer conventions reverse model rankings: Qwen2.5-VL-7B and Qwen3-VL-4B tie under English names but differ by +39.0 points under pixels and -54.5 under normalized coordinates, each model leading in the convention its own technical report says it was trained in.
- The mispaired-name readership test separates conventions a model reads (normalized 0-999 at 4.05x chance) from those it avoids (hue angles at 0.30x), though a pre-registered test showed readership does not predict accuracy.
- Five models partition one calibrated CIELAB color wheel into five different stable vocabularies, so a fixed set of color answer words is not a neutral instrument across models.
Strengths
Paired convention-swap design cleanly isolates readout from capability
Image, question, candidate set, order and output format are held fixed; the digit-emission cost is measured separately at 1 point (Table 1), so the 30.1-point word-to-hue-angle drop (McNemar p=8.7e-33, n=519) cannot be attributed to output formatting.
Unusual transparency: pre-registered plan including failed predictions
Hypotheses, endpoints, exclusions and failure conditions were registered before code existed (Section 3.6); the held-out drop (99% to 84%) and the readership accuracy prediction failure (Section 6) are reported rather than buried.
Robustness of the headline effect is extensively checked
Mispaired-name readership test is a genuinely new diagnostic
Candid self-audit of scoring failures (Section 8)
Weaknesses
Narrow external validity
Frontier evidence is thin and the null may be a floor effect
Instrument mismatch between local and frontier scoring
The proposed readership metric fails its own out-of-sample test
Code not yet publicly available
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a measurement problem in vision-language model (VLM) evaluation: the surface convention of multiple-choice answers (English phrases, hue angles, pixel pairs, normalized coordinates) is treated as a neutral instrument, and the author shows it is not. Central claims: (1) on 200 COCO val2017 photographs, Qwen3-VL-4B scores 68.5% under English cell names versus 20.0% under absolute pixel coordinates on a 9-option locate task (chance 11.1%), a paired gap of -48.5 points (exact McNemar p=1.2e-21, 95% bootstrap [-56.5, -40.0], Table 2); (2) the cost is answer-side — the same coordinate as question context costs 3.5 points and is not significant (p=0.065); (3) conventions reverse rankings (Table 4: Qwen2.5-VL-7B vs Qwen3-VL-4B is -0.5 under names, +39.0 under pixels, -54.5 under normalized, the latter two p<=1.2e-30); (4) a mispaired-name 'readership' test separates conventions that are read from those avoided; (5) five models partition one CIELAB color wheel into five vocabularies (Table 5). Claims are scoped honestly: the author explicitly disclaims free-form box-regression conclusions and labels post-hoc analyses as such.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 3 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a 'readout limit' in vision-language models according to this paper?
It is a score limit caused by how a benchmark writes its answer choices (for example pixel coordinates instead of plain English phrases) rather than by what the model can actually see or know. The paper shows the same question can yield 68.5% or 20.0% accuracy depending only on how the options are described, with 11.1% chance.
Which models were tested in this study?
Five open models run locally (Qwen3-VL-4B at 4- and 8-bit, Qwen2.5-VL-7B, InternVL3-2B, LLaVA-1.5-7B, SmolVLM-2B) plus two frontier models via API (Gemini Flash and GPT-4o, as served on 20 September 2026). The real-image task used 200 COCO photographs.
Does this paper mean spatial benchmarks with coordinate answers are invalid?
No. The author argues coordinates on the answer side should be reported together with the coordinate system used, that a convention swap (describing the same options a second way) should accompany results, that parse failures should be separated from wrong answers, and that answer vocabularies should be calibrated per model — not that coordinate benchmarks are meaningless.
What is 'readership' and what does it measure?
Readership comes from a mispaired-name test: each coordinate is glossed with a wrong name, and readership is the share of answers following the coordinate relative to chance. It flags conventions the model actually reads (normalized 0-999: 4.05x chance) versus ones it avoids (hue angles: 0.30x). Notably, a pre-registered test showed readership does not predict accuracy for new conventions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is only available as caption text; per-model bar values were taken from the caption and Table 6, not from the rendered figure.
- Frontier API results are pinned to a serving date (20 September 2026) and cannot be exactly re-verified later.
- Code, logs and the analysis plan are promised but not yet public, so audit and pre-registration claims rest on the paper's word.
- The arXiv abstract is truncated mid-sentence, but the embedded full text runs complete through references and appendices, so coverage loss is minimal.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.