Learning to Refer from Estimated Listener Gaze
cs.CL
Quick verdict
Gaze-scanpath rewards make VLM speakers more pragmatic; strong ablations and human eval, but no significance tests and weak cross-backbone generalization.
Summary
The paper proposes finetuning vision-language models (VLMs) as referring-expression speakers using a novel learning signal: estimated listener gaze. The authors first train Molmo-REC-Gaze, a Molmo-7B-D-based incremental listener that predicts word-aligned fixation sequences during reference comprehension, outperforming the ART baseline on RefCOCO-Gaze (DTW path distance 39.9 vs 55.9; endpoint accuracy 84.4% vs 66.7%). They then optimize a speaker policy with REINFORCE, converting sampled utterances' estimated scanpaths into token-level rewards via four schemes (SeqLPHit, SeqAnyHit, BeforeFirstHit, Shaping), compared against non-gaze listeners (binary success, iterative prompting). With human listeners (633 filtered Prolific participants), gaze-trained speakers shrink average length from 15.4 to 4.0-9.5 words while keeping or improving success (75.2% baseline up to 80.0% with Gaze-SeqAnyHit); non-gaze listeners reach higher success (86%) only via verbosity (up to 24.4 words). Strengths include extensive ablations (KL penalty, length penalty, listener variants, SFT comparison, extra backbones) and behavioral metrics (time-to-target, early clicks). Limitations: no formal significance tests despite 'significant' claims; results rest on Molmo-7B, with gains not cleanly transferring to LLaVA-1.5/PaliGemma; the SFT baseline using 8.8k RefCOCO examples outperforms all RL variants (dπ 0.59 vs 0.81), and the gaze listener is itself less accurate on machine-generated text, a confound the authors acknowledge.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Finetuning VLM speakers with token-level rewards derived from estimated listener gaze scanpaths yields more pragmatically optimal referring expressions than training from outcome-only communicative success.
- With human listeners, gaze-trained policies cut average reference length from 15.4 to as low as 4.0 words while maintaining or raising success from 75.2% up to 80.0% (GAZE-SEQANYHIT).
- Non-gaze listeners drive higher success (up to 86.0% with REC-SUCCESS) but at severe verbosity costs (up to 24.4 words with REC-SEQANYHIT), exposing an accuracy-verbosity tradeoff in current VLMs.
- GAZE-SHAPING is the only machine speaker whose referential strategy profile (predominantly spatial reference) resembles human references, without explicit stylistic supervision.
- A Molmo-7B-D-based incremental gaze listener (Molmo-REC-Gaze) estimates human scanpaths better than prior ART (DTW path distance 39.9 vs 55.9; end-point accuracy 84.4% vs 66.7%).
Strengths
Novel dense learning signal from incremental comprehension
Instead of the single outcome feedback of prior interaction-based REG finetuning (Gul & Artzi 2024; Tang et al. 2024), the paper converts word-aligned gaze fixations into per-token rewards (Sec 5), motivated by psycholinguistic work on speakers monitoring listener visual search (Clark & Krych 2004; Jara-Ettinger & Rubio-Fernandez 2022).
Unusually thorough ablations and controls
KL-penalty ablation showing entropy collapse and 1-2 token lengths without it (App C.1), length-penalized REC-SUCCESS control (C.2), final-gaze-point listener and RefCOCO-finetuned listener controls isolating incrementality from data exposure (C.3.1-3.2), SFT comparison (C.4), and cross-backbone tests (C.5).
Human evaluation with behavioral process metrics
Honest error analysis of reward hacking and artifacts
Gaze predictor itself advances the state of the art
Weaknesses
No formal statistical testing despite significance claims
Single-backbone dependence with failed generalization
SFT baseline dominates the proposed method
Gaze-estimator confound on machine-generated text
Narrow domain and scale
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Learning to Refer from Estimated Listener Gaze
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does learning from estimated listener gaze mean in this paper?
The authors train a vision-language model (Molmo-REC-Gaze) to predict where a human listener's eyes would move word-by-word while interpreting a referring expression. During speaker training, sampled utterances are fed to this gaze model, and the predicted fixation sequence is converted into per-word rewards — favoring utterances that guide the simulated gaze to the target quickly — so a VLM speaker learns to be concise and unambiguous without human feedback in the training loop.
How much shorter and more accurate are the generated referring expressions?
With human listeners, the base Molmo model produces 15.4-word references at 75.2% success. Gaze-trained models reduce length to 4.0 words (GAZE-SHAPING, 75.5% success) or raise success to 80.0% (GAZE-SEQANYHIT, 9.5 words). Models trained with a plain machine listener reach 86% success but generate up to 24.4-word references; human-written references achieve 92.5% at 3.4 words.
Is code or data available for this paper?
Yes. Code and model weights are released at github.com/Berkeley-NLP/reg-from-gaze (footnote 1). Training data derive from public sources: RefCOCO-Gaze (gaze scanpaths), COCO-2014 images for speaker training, and RefOI/RefCOCO test sets for evaluation; the RefCOCO-Gaze test split is not public, so gaze-prediction numbers use its validation set.
Does the method work on vision-language models other than Molmo?
Not cleanly. Appendix C.5 reports that with LLaVA-1.5-7B and PaliGemma-3B under unchanged hyperparameters, gains are unstable or negative — for example GAZE-SHAPING lowers LLaVA-1.5 accuracy from 28.3% to 20.5%. The authors attribute this to missing model-specific tuning (especially the KL coefficient), so the headline results are demonstrated only on Molmo-7B-D.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure content (Figs 1-2, 5-11) is partially garbled by LaTeX-embedding extraction; numeric claims rely on tables, which extracted cleanly.
- Appendix D.3 (qualitative behavior) heading appears without body text in the extraction; that analysis could not be assessed.
- Human-evaluation power assessment is inferred from stated sample sizes; raw per-item response data were not available.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.