GazeRefine: Expert Gaze as a Test-Time Prompt for Training-Free Medical Image Segmentation

Summary

GazeRefine is a training-free segmentation framework that uses clinician eye-gaze as an inference-time prompt for zero-shot medical image segmentation. Duration-weighted fixations are converted into foreground and background priors that initialize prototypes in the frozen feature space of DINOv3 (ViT-L/16); the prototypes are then iteratively refined through foreground-background confidence suppression, kNN affinity propagation among patches, and blending back toward the initial gaze anchors (lambda=0.5, T=5, k=5). The pipeline uses no segmentation masks, gradient updates, adapters, or prompt encoders, and thresholding the upsampled confidence map at 0.5 yields the final mask. On gaze-annotated Kvasir-SEG polyps it reaches 88.10±0.14 Dice, close to fully supervised nnU-Net (88.41±0.47) and above zero-shot gaze-guided GazeMedSAMv2 (85.19±0.13). On NCI-ISBI prostate MRI it obtains 80.28±0.11, clearly below GazeMedSAMv2 (85.62±0.10), which the authors attribute to weaker foreground-background separability in general-purpose features. The ablation shows background cleaning is the dominant component — removing it collapses Dice to 28.70/13.50 — while kNN propagation mainly helps prostate MRI. Limitations include Dice-only evaluation, unexplained tiny standard deviations, a per-dataset Gaussian spread hyperparameter (sigma=2.0 vs 0.5) raising test-set tuning concerns, mis-cited VAM baselines, and no sensitivity or anchoring ablation.