Emergent Goal-Directed Attention in Large Vision-Language Models
cs.CV · cs.AI · cs.CL
Quick verdict
Claims zero-shot VLMs align with human goal-directed attention on 4,887 scenes; crossover evidence is strong and multi-level, with contamination caveats.
Summary
The paper asks whether general-purpose vision-language models (VLMs) can exhibit human-like, goal-directed attention without any training on gaze data. Two off-the-shelf models, Qwen3-VL-32B-Thinking and Gemma-4-26B-A4B-it, were prompted under visual-search and free-viewing instructions on 4,887 naturalistic scenes from COCO-Search18 and COCO-FreeView, and their emitted points were compared with human fixations via correlation of priority maps. Both models showed a robust crossover: predictions aligned better with human fixations under matching than mismatching goals (e.g., Qwen target-present search r=.73 vs .15). The crossover survived in 2,413 target-absent scenes, where simple visual grounding cannot explain alignment, and reappeared in decoder-layer attention readouts. Qwen's thinking traces were semantically closer to the target word under search and to visual-prominence terms under free viewing. Scanpath analyses showed near-zero order alignment during search, attributed to center-bias and holistic-inference differences. Zero-shot correlations exceeded DeepGaze II on search but fell short of a supervised model trained on the same fixations. Limitations include only two frontier-scale models, attention-layer bands tuned on 54 pilot images, possible COCO training-data exposure, and ad hoc semantic anchors. Data, code, and model outputs are public.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Off-the-shelf VLMs produce spatial priority maps that align better with human fixations when model and human goals match (crossover effect) across 4,887 scenes.
- The crossover persists on 2,413 target-absent scenes, indicating alignment draws on scene-semantic knowledge rather than localizing a visible target.
- Goal-related alignment is also present in decoder-layer attention readouts of both models.
- Qwen's thinking traces are semantically closer to target words during search and to visual-prominence terms during free viewing.
- Zero-shot VLM correlations are competitive with gaze-trained models (exceeding DeepGaze II on search) but VLMs fail to predict human fixation order during visual search.
Strengths
Internally controlled crossover design
The same 4,887 images are evaluated under both goals for models and humans, so low-level salience and image-specific confounds cancel in the goal-matching comparison; the Qwen target-present interaction is enormous (F(1,2473)=12,901.70, etaG2=.48) with search-search r=.73 vs search-free r=.15.
Target-absent condition cleanly rules out visual grounding
On 2,413 scenes with no visible target, models must predict where an absent target would plausibly be; the crossover persists (Qwen search-search r=.45 vs search-free r=.19; interaction etaG2=.174), so alignment must reflect scene-semantic priors rather than localizing a visible object.
Multi-level converging evidence
Honest reporting of negative and weak results
Strong reproducibility assets
Weaknesses
Generality limited to two frontier-scale models
Attention layer bands tuned on pilot images
Unquantified training-data contamination
Human-model task and map-construction mismatches unexamined
Ad hoc semantic anchors in the thinking-trace analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Emergent Goal-Directed Attention in Large Vision-Language Models
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did the VLMs beat trained saliency models like DeepGaze II?
On matched-goal search tasks, yes by correlation: zero-shot Qwen/Gemma reached r=.73/.70 on target-present search and .45/.40 on target-absent search, versus .24/.30 for DeepGaze II on the same data. A supervised model trained directly on these fixations was still higher (.68/.54/.59), and the authors call the comparison preliminary rather than systematic.
What is the crossover effect in this paper?
Model predictions made under a search instruction matched human search fixations better than human free-viewing fixations, and predictions made under a free-view instruction did the reverse. This shows the model reorganized its spatial priorities according to the task goal, as humans do, on the very same images.
Could the models just have memorized the COCO images during training?
Exposure cannot be ruled out because training data is undisclosed and COCO is widespread online. The authors argue that goal-dependent differences in predictions cannot come from memorization alone, and the target-absent results require scene knowledge, but no direct contamination probe was performed.
Are the code and data available?
Yes. Model outputs are on Zenodo (record 22151952), analysis code on GitHub (HanZhang-psych/VLM-goal-directed-attention), model weights on HuggingFace, and the human fixation data (COCO-Search18, COCO-FreeView) is publicly available from the original authors.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure images are not extractable from the PDF text; assessments of Figures 1-8 and S1-S7 rely on captions and in-text statistics only.
- Per-category results shown in Figures S5/S6 appear only as captions in the provided text; the underlying category-level values are not present.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.