(How) Do MLLMs Report Bistable Images Like Humans?
Ryota Takatsuki, Tomoki Doi, Amane Watahiki, Anil K. Seth, Hitomi Yanaka
cs.CV · cs.AI
Quick verdict
Claims LLaVA MLLMs show human-like bistable reporting with identifiable mechanisms; well-controlled and largely convincing, though statistics are thin.
Summary
The paper asks whether multimodal large language models (MLLMs) report ambiguous images the way humans do, and what internal computations support that behavior. Focusing on the duck-rabbit illusion, the authors test five LLaVA-family models (plus four non-LLaVA models for behavioral checks) on two report-level properties: modulability, whether reports shift under bottom-up visual manipulations (rotation, red-circle cueing) and top-down linguistic cues (letter prefixes, semantic hints), and exclusivity, whether the model commits to a single interpretation. To reduce memorization confounds, they generate 50 novel synthetic ambiguous images (Visual Anagrams), filtered by a VLM judge, with matched two-animal and single-animal controls. Behaviorally, rotation reliably reverses the preferred interpretation, red-circle placement aligns qualitatively with human fixation-density data, linguistic cues bias reports only when the image supports the cued percept, and reports stay predominantly exclusive for ambiguous stimuli while two-animal controls are enumerated. Mechanistically, direct logit attribution and logit lens localize interpretation evidence in image tokens across middle-to-late layers; query patching shows top-down cues act mainly through prefix-to-output transfer rather than image-attention queries (about 3% of the effect); and image-token patching ties exclusive reporting to object-count encoding, with one model as an exception. Main limitations: descriptive statistics only, reliance on a proprietary judge model, and no switching dynamics over time.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLaVA-family MLLM reports on the duck-rabbit and synthetic Visual Anagrams are systematically modulable by rotation and red-circle cues (bottom-up) and by prefix and semantic linguistic cues (top-down), with linguistic effects bounded by visual evidence for the cued percept.
- Reports on ambiguous stimuli are predominantly exclusive (about 10% or less multi-animal beam mass near the boundary), whereas matched two-animal controls are enumerated (>=98% multi-animal mass).
- Exclusive reporting covaries with single-object (count '1') encoding across models and is causally linked to it by image-token patching, though Llama3-LLaVA-Next-8B shows exclusivity without strong count encoding.
- Bottom-up modulation changes the balance of interpretation evidence in image tokens, while top-down cues propagate from prefix to output token over layers 15-20 with early-MLP involvement rather than via image-attention queries (query patching explains only ~3% of the top-down effect).
Strengths
Memorization-controlled stimulus design
50 synthetic Visual Anagrams generated with DeepFloyd IF and accepted only when a GPT-5-nano judge names the correct animal in >=4/5 queries per view (§A.1.1); key behavioral patterns replicate on these novel stimuli, directly answering the Ullman (2024)-style confound that models report famous illusions from memory.
Matched controls isolate ambiguity from enumeration ability
Style-matched two-animal and single-animal controls (§A.1.1): two-animal controls receive >=98% multi-animal beam mass (Fig 5c, Table 2) and resist cue induction (0.03-0.05 cued-percept mass vs 0.23-0.43 on ambiguous stimuli, §B.3.2), showing exclusivity is specific to ambiguity rather than a general listing deficit.
A falsified hypothesis strengthens the mechanistic contribution
Behavior-to-mechanism arc with cross-model covariation
Generality and robustness checks
Weaknesses
No inferential statistics
Qualitative human comparison
Judge-in-the-loop measurement
Selection filter and single-model mechanistic depth
Narrow scope of 'like humans'
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether MLLMs report bistable images 'like humans', operationalized along two dimensions: modulability (sensitivity of the reported interpretation to bottom-up manipulations — rotation and red-circle cueing — and top-down linguistic cues — letter prefixes and semantic hints) and exclusivity (committing to a single interpretation). Stimuli are the canonical Harper's duck-rabbit plus 50 synthetic Visual Anagrams (DeepFloyd IF; GPT-5-nano judge, 50/100 accepted with >=4/5 correct per view) and matched two-animal/single-animal controls. Models: five LLaVA variants sharing CLIP ViT-L (1.5-7B/13B, v1.6-Vicuna-7B, v1.6-Mistral-7B, Llama3-LLaVA-Next-8B) plus four non-LLaVA behavioral checks. Each abstract claim — human-consistent modulation, predominant exclusivity, image-token competition, distinct modulation pathways, and object-count encoding — maps to a specific experiment, and the mechanistic story is honestly hedged: the initial query-mediated hypothesis is tested and rejected.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do multimodal AI models see the duck-rabbit illusion like humans?
According to this paper, LLaVA-family models match humans in two narrow respects: their reported interpretation shifts with image rotation, red-circle visual cues, and linguistic hints much as human reports do, and they report only one animal at a time for ambiguous images while listing both animals for a genuine two-animal picture. The paper does not claim models experience perceptual switching over time.
How did the authors prevent models from simply memorizing the famous duck-rabbit image?
They generated 50 novel synthetic ambiguous images ('Visual Anagrams') that show a duck upright and a rabbit when rotated, accepted only those where a VLM judge (GPT-5-nano) named the correct animal in at least 4 of 5 queries per view, and replicated the key behavioral patterns on these previously unseen stimuli.
What internal mechanism explains the models' single-interpretation reports?
Mechanistic analyses (direct logit attribution, logit lens on image tokens, activation patching) show that competing duck/rabbit evidence lives in image tokens read out in mid-to-late layers, and that a small subset of image tokens encodes object count; patching these tokens shifts both '1 vs 2' answers and whether the model reports one or both animals. One model (Llama3-LLaVA-Next-8B) stays exclusive without strong count encoding, so the link is not universal.
Is the code and data for this paper available?
Yes. The paper states that code and data are available at https://github.com/rtakatsky/mllm-bistable-images, and all nine evaluated models are public checkpoints (LLaVA variants, Qwen2-VL, SmolVLM, IDEFICS2, InstructBLIP).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with marked '[...sampling gap...]' sections: parts of §3.2.1, the end of §4.2.2 through §4.3 (object-count patching, Figure 12), and parts of Appendix C were skipped; judgments on those parts rely on surviving fragments (Figure 12 caption, C.1.5, C.2).
- Figures are referenced in captions but their pixel content is not extractable from text; all numeric readings in this review come from values stated in the body text, captions, and Table 2.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.