What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies

Summary

This paper reframes distractor-induced failures in visuomotor imitation learning as "conditional visual grounding" errors: the visual entity that matters changes with manipulation phase (object during picking, receptacle during placement) and, in one task, with observed object state. Using ACT trained on 100 scripted demonstrations per task, the authors systematically introduce color- or shape-matched distractor objects and receptacles and localize failures with conditional metrics. In Task 2, three color-matched object competitors collapse P(pick) to 39.2% while conditional lift and placement stay near 94–96%; two shape-matched receptacles leave picking intact but drop conditional placement to 33.9%. Three interventions — copy-paste distractor augmentation, phase-dependent attention regularization, and positionless visual prompting with a learned phase predictor — raise success under full mixed distractors from 39.5%/14.0% to 94.5%/88.5% in simulation and from 0/20 to 13/20 and 12/20 on a physical UR3e. A representation analysis shows augmentation alone yields stronger distractor invariance but weaker container geometry and worse conditional placement (66.3% vs 100.0%), arguing robustness needs invariance plus preserved task geometry. A prompt-swap probe (0–1% to 35–91% distractor selection) supports a causal reading. A small π0.5 case study shows the same pattern (routing 5/10 to 10/10 when a cue is restored). Limitations: no statistical testing, 20 hardware trials per cell, asset-specific cue hierarchy, non-equivalent VLA conditions, and no code release.