Same evidence, different judgments: Evidence noncommutative in vision/speech-text conflicts
Zhuoyun Li, Boxuan Wang, Xiaowei Huang, Xiaowei Huang, Yi Dong
cs.AI
Quick verdict
Same-evidence order swaps shift multimodal judgments toward image/speech; a well-controlled empirical claim, though "noncommutativity" slightly outruns two-element scope.
Summary
When an image or speech recording conflicts with accompanying text, multimodal LLMs often favor text; the authors argue such measurements entangle modality preference with evidence position. They swap only the positions of the two sources, keeping instructions and content fixed. Across GQA/VG, VQAv2/COCO, FSC and SLURP (240 examples each) and six open-weight models (3–12B), placing the image or recording last shifted answers toward it by +9.8 to +28.1 pp in all twelve model–dataset combinations, with paired log-odds swap-score CIs (paired bootstrap) all excluding zero. Subtracting an unrelated-text reordering baseline (D = ∆A− − ∆A0) leaves +9.4–28.8 pp (vision) and +10.4–24.0 pp (speech). The effect survives source-priority instructions, alternative wording, free-form responses, and (variably) inserted tokens; text recency stays an uneliminated contributor. A reproduction of "Words or Vision" shows instruction placement can reverse the measured direction. Limitations: only 3–12B open models on A/B selection, no code-release statement, one unreconciled baseline across Tables 1/2, and "noncommutativity" remains two-element.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Swapping the positions of perceptual and conflicting textual evidence shifts attachment-answer selection by +9.8 to +28.1 pp in all twelve vision/speech model–dataset combinations, with all 95% CIs excluding zero.
- The attachment-last advantage is conflict-specific: after subtracting the unrelated-text reordering baseline (D = ∆A− − ∆A0), effects remain +9.4–28.8 pp (vision) and +10.4–24.0 pp (speech).
- Instruction placement is a protocol confound: under the original Words or Vision setup moving the image last changed selection by −28.6 pp versus +9.5 pp with fixed framing (+38.1 pp difference), enough to reverse the measured direction of text bias.
- The order effect persists under source-priority instructions, alternative attachment wording, free-form responses, and inserted filler tokens, though text recency remains an uneliminated contributor.
- Fixed-order conflict benchmarks (e.g., MCR-BENCH) leave the gains from placing perceptual evidence last unmeasured.
Strengths
The paired-swap protocol cleanly separates order from instruction placement
Instructions sit before both sources and question/options after them (Eq. 1), so only P and T exchange; A/B option positions are counterbalanced in every condition (Sec. 3.1, nine input conditions), attacking the option-letter bias that plagues this literature [9,33].
The order effect is uniform and statistically reported
Table 1: ∆A− spans +9.8 to +28.1 pp across all twelve model–dataset combinations, and mean log-odds swap scores C have 95% CIs excluding zero everywhere (e.g., Qwen2.5-VL on GQA/VG +16.7 pp, C = 0.65 [0.50, 0.80]; Idefics3 on VQAv2/COCO +28.1 pp, C = 4.77 [4.40, 5.14]).
Controls establish conflict-specific, not generic, reordering
Section 4.5 turns a critique into a mechanistic demonstration
Unusually broad modality and robustness coverage for a 4-page paper
Weaknesses
Scope of the generalization is narrow relative to the framing
An unreconciled cross-table inconsistency
Constructed text conditions are never externally validated
Statistical reporting is thin on details and multiplicity
The mechanism is left open and text recency is a live confound
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This ICASSP-style short paper (Univ. of Liverpool; arXiv 2609.26986, cs.AI) claims **cross-modal evidence noncommutativity**: in vision/speech–text conflict settings, exchanging the positions of perceptual evidence P and text T changes MLLM judgments. Concretely, placing the image/recording **after** conflicting text raises attachment-answer selection by **+9.8 to +28.1 pp across all twelve model–dataset combinations** (Table 1), with paired log-odds swap scores whose 95% CIs all exclude zero (e.g., Idefics3 on VQAv2/COCO: +28.1 pp, C = 4.77 [4.40, 5.14]). The paper's second contribution is diagnostic: earlier text-bias studies (Words or Vision [4], MCR-BENCH [10]) compared orders while the task framing and warnings moved with the text, confounding position with instruction placement.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'evidence noncommutativity' in multimodal LLMs?
It means the order of conflicting evidence changes the model's judgment even though the evidence content is identical: the same image-plus-text pair, swapped so the image comes last, produces different answers. The paper shows placing an image or speech recording after conflicting text shifts answers toward it by +9.8 to +28.1 percentage points across six models and four benchmarks.
Why do earlier studies report text bias in vision-language models, while this paper reports an image/speech-last advantage?
The authors argue the discrepancy is a protocol artifact: in prior setups (e.g., 'Words or Vision'), the task instructions, warnings, and conflicting text moved together when order was swapped, confounding position with instruction placement. Their reproduction shows the original protocol yields −13.7 pp for image-last on Qwen2.5-VL-3B, while fixing the instructions flips this to +17.7 pp.
Do multimodal models always follow the evidence that appears last in the prompt?
Not unconditionally. The paper's claim is that under conflicting text, evidence placed later gains weight — by +9.8 to +28.1 pp in all twelve tested model–dataset combinations, with 95% confidence intervals excluding zero. The effect persists with priority instructions, alternative wording, and free-form responses, but text recency remains a contributor the authors could not rule out.
Is code available for this paper?
No. The extracted text contains no code or data release statement, and prompt templates and the constructed conflicting/unrelated text sets are only partially quoted, so the full configuration cannot be reconstructed from the paper alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review based on embedded text extraction; Figures 1–3 (attention histogram, ablation bars) and Fig. 2's design panel are described but not visually analyzable.
- Table 2's column layout was partially garbled by extraction; numbers were cross-checked against in-text prose.
- The reference list (items 14–22, author self-citations) is included but was not central to the assessment.
- arXiv page numbering/versioning beyond v1 was not verifiable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.