Reasoning Instructions Can Break Answer Decoding in Vision--Language Models
Zeyan Li, Siyuan Qiu, Jianfeng Xu
cs.CL · cs.AI
Quick verdict
A convincing evaluation-methodology study showing CoT-prefix scoring, reading label logits after a reasoning request, understates VLM accuracy by 35.28 points on ScienceQA; evidence is strong but coverage narrow and unaccompanied by significance testing.
Summary
The paper identifies and names "CoT-prefix scoring," an evaluation convention in which a scorer appends a continuation-inducing cue yet reads answer-label logits immediately. On ScienceQA, Qwen2.5-VL-7B collapses from 80.76% to 45.48% accuracy, and across five option-content permutations 93.54% of predictions select the first option. Condition-matched linear probes and free generation recover 78.94% and 75.24% respectively, showing the answer survives inside the states and only the immediate readout fails. Diagnostics show probability mass migrates toward continuation tokens while answer information remains linearly accessible in late layers; fine-tuning only the output projection restores 70.90%. The effect reproduces on AI2D and MMMU, reverses on MMBench, and weakens with more choices. Main limitations: coverage limited to multiple-choice benchmarks and two VLM families, no statistical significance testing, and probes establish decodability rather than causal use by the model itself.
Review complete
Assessment at a glance
0
risk flags identified
● high 0
● medium 0
● low 0
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CoT-prefix scoring, appending a continuation-inducing cue but scoring answer-label logits immediately, collapses multiple-choice accuracy (Qwen2.5-VL-7B: 80.76% to 45.48% on ScienceQA) and drives 93.54% of predictions to the first slot.
- The answer survives the prefix: condition-matched probes, free generation, and LoRA retraining of only the vocabulary projection each recover most of the lost accuracy, localizing the failure to the immediate readout.
- Mechanistically, the suffix shifts probability mass toward continuation tokens while answer identity remains linearly decodable in late layers, so renormalizing label logits does not restore the original decision.
- The collapse is not universal: it recurs on AI2D, MMMU-10 and LLaVA-1.5-7B, reverses on MMBench (+5.80 points), and weakens as the number of displayed choices grows.
- Evaluations should align the requested output event with the scored output event and report the exact prompt/scorer interface.
Strengths
Cleanly isolated conditions
Direct vs. CoT-prefix differ only in the terminal instruction, isolating the suffix; Free CoT isolates the generation event. §2 defines the event mismatch formally before any results.
Four mutually reinforcing controls
Content permutation (93.54% first-slot rate), matched probes (78.94% recovery), free generation (75.24%), full-string likelihood control (reversal to 55.41%), and LoRA-on-output-projection (45.48%→70.90%) jointly localize the failure to the immediate readout.
Mechanism is measured, not asserted
Honest, non-universal bounding
Weaknesses
No inferential statistics
Diagnostic subsets unjustified
No release statement
Position vs. label bias unresolved
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Reasoning Instructions Can Break Answer Decoding in Vision-Language Models
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 0 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CoT-prefix scoring?
It is an evaluation shortcut where the evaluator appends a continuation-inducing instruction (e.g., "Let me think step by step.") to a multiple-choice question, then reads the answer directly from next-token label logits at the very next position, without letting the model generate any reasoning. The prompt requests a rationale while the scorer reads a bare answer label.
Why does CoT-prefix scoring hurt VLM accuracy?
Because a prompt that invites reasoning shifts next-token probability mass toward continuation tokens rather than answer labels, and within the label set predictions concentrate on the first slot (93.54% on ScienceQA). The paper stresses the answer often survives internally; what breaks is the immediate readout, not the model's knowledge.
Does this mean the model does not know the answer?
No. Condition-matched linear probes recover 78.94% accuracy from the same hidden states, free generation reaches 75.24%, and LoRA-tuning only the output projection restores 70.90% versus 45.48% under CoT-prefix scoring. This is evidence of readout failure, not knowledge loss.
Does the effect occur in all VLMs and datasets?
Not universally. It recurs on AI2D (up to -54.02 points), MMMU-10, and LLaVA-1.5-7B, but reverses on MMBench (+5.80 points for Qwen2.5-VL-7B), and it weakens as the number of displayed choices grows.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are rendered as text; visual details of layer profiles and histogram shapes could not be verified.
- The PDF extraction shows section 3 heading text but no heading formatting; tables were parsed inline and some column alignments were inferred.
- No code/data availability statement appears in the provided text; possibly truncated.
- MMMU-10 subset selection (286 items) is not defined in the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.