Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts
Yen-Ting Piao, Shu-Yun Chen, Chin-Hui Chu, Chun-Wei Chen, Shih-Yun Shan Kuan, Hung-yi Lee, Yun-Nung Chen
cs.CL · cs.AI · cs.CV · cs.SD
Quick verdict
Solid tri-modal benchmark disentangling modality from evidence-form bias; credible results, but propositional evidence is confounded with TTS/rendering.
Summary
The paper introduces Tri-PvP, an 8,000-sample benchmark probing modality bias in omni-modal LLMs under three-way conflicts among image, audio, and text, where the image and audio channels each take either perceptual form (natural photos or recordings) or propositional form (rendered text images, TTS utterances), yielding four matched evidence-type conditions across four everyday domains. Evaluating five OLLMs (Qwen2.5-Omni, MiniCPM-o 4.5, Qwen3-Omni, Gemma 4, Gemini 3 Flash) with an LLM judge validated at 97.1% agreement on 800 stratified samples, the authors find image bias dominant in 18 of 20 model-condition bars, audio bias consistently weakest, and conflict-aware (NO_BIAS) answers peaking under fully propositional conditions (60.1% for Qwen2.5). The claimed asymmetry — perceptual preference in vision, propositional in audio — holds for most models, with Gemma4 a vision-side exception under the default ordering (74.4% vs 60.3%). Layer-wise probing shows bias linearly decodable from early layers, and contrastive decoding on Gemma4 cuts image bias (74.9% to 38.8% in PropI-PropA) at the cost of residual text bias and roughly a one-point OmniBench drop. Limitations: propositional evidence conflates evidence form with signal realization (TTS, rendered text), no significance testing despite the matched counterfactual design, and mitigation tested on a single model.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing tri-modal modality-bias benchmarks confound modality bias with evidence-form bias because the perceptual/propositional mix is unmatched across modalities.
- Across five OLLMs and four matched evidence-type conditions, image bias dominates (18 of 20 bars) and audio bias is consistently the weakest single-modality bias.
- Models show stronger bias toward perceptual evidence in vision but propositional evidence in audio, with the vision-side pattern reversed for Gemma4 under the default ordering.
- Modality bias is linearly decodable from early intermediate hidden states, before any token is generated, with model-specific layer-wise dynamics.
- Contrastive decoding partially reduces image bias (74.9% to 38.8% on Gemma4 PropI-PropA) without parameter updates but induces a residual text bias.
Strengths
Genuine confound identified and controlled
The perceptual/propositional factorial axis (Sec 3-4) with matched counterfactuals — every label triple instantiated in all four evidence-type conditions with asset reuse (Sec 4.1) — directly fixes the confound the paper diagnoses in CMM and MMA-BENCH, and supports clean within-triple comparisons.
Multi-level evidence chain
Behavioral distributions (Fig 3), six-way input-ordering robustness (Tables 1-2, 5-6), representation-level probes (Fig 4), and a decoding-time intervention with a general-competence control on OmniBench (Table 11) make the claims harder to dismiss as prompting artifacts.
Unusually careful evaluation hygiene
NO_BIAS is not treated as monolithic
Reproducibility and licensing
Weaknesses
Evidence form is confounded with signal realization
The 'systematic asymmetry' is overstated
No inferential statistics despite a matched design
Mitigation is thin and the competence cost is uneven
Probing targets and NO_BIAS semantics are coarse
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Tri-PvP (arXiv 2609.06011) is a benchmark paper on modality bias in omni-modal LLMs. Its central thesis: existing tri-modal conflict benchmarks mix perceptual evidence (photos, real recordings) and propositional evidence (declarative statements) within modalities without matching the mix across modalities, so any measured modality bias is confounded with evidence-form bias. The authors build an 8,000-sample benchmark over four everyday domains (animal, emotion, environment, music) with four matched evidence-type conditions and evaluate five OLLMs (Qwen2.5-Omni, MiniCPM-o 4.5, Qwen3-Omni-30B-A3B-Thinking, Gemma 4 E4B, Gemini 3 Flash). Three headline claims: (1) image bias dominates (18 of 20 model-condition bars, Fig 3); (2) a 'systematic' asymmetry — perceptual preference in vision, propositional in audio; (3) bias is linearly decodable from early layers and only partially mitigable by contrastive decoding (CD).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Tri-PvP?
Tri-PvP is an 8,000-sample benchmark that creates three-way conflicts among an image, an audio clip, and a text statement, where the image and audio can each be perceptual (a real photo or real-world recording) or propositional (a rendered text card or a TTS-spoken sentence). It measures which input source omni-modal LLMs rely on when the sources disagree.
What is the difference between perceptual and propositional evidence in this paper?
Perceptual evidence is direct sensory experience, such as a photo of a dog or a recording of barking. Propositional evidence is a declarative statement that is true or false, such as the written or spoken sentence 'this is a dog'. Borrowed from epistemology, this distinction is used as an experimental axis, and the paper shows LLMs treat the two forms differently within the same modality.
What are the main findings of Tri-PvP?
Across five omni-modal LLMs, answers follow the image in 18 of 20 model-condition settings, audio is consistently the least trusted source, models lean toward perceptual visual evidence but propositional audio evidence (with exceptions such as Gemma4 under the default prompt order), and a decoding-time intervention reduces image bias only partially while adding a text-side side effect.
Is the Tri-PvP dataset and code available?
Yes. The paper points to a public GitHub repository (MiuLab/Tri-PvP) and releases the benchmark under CC BY-NC 4.0 for non-commercial research. Underlying perceptual assets come from licensed public datasets (ImageNet, ESC-50, KDEF, CREMA-D, Open Images V7, Medley-solos-DB) that retain their original terms.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 3 and Figure 4 values appear in the extraction as run-on digit/axis streams; Figure 3 statistics were cross-verified against the numeric Tables 5-9 (they agree), while probing accuracies exist only qualitatively in the text (e.g., 'approximately 80%') with no per-layer table anywhere in the paper text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.