Proprioception-Anchored Cross-Modal Pretraining for Zero-Shot Sim-to-Real Contact-Rich Assembly

Summary

The paper proposes PACE, a multimodal encoder for zero-shot sim-to-real transfer in contact-rich assembly. The core observation is that proprioception—joint positions in a shared calibrated coordinate system and velocities computed by finite differencing—is approximately consistent across simulation and hardware, whereas visual and force/torque (F/T) observations differ. PACE pretraining makes visual and F/T temporal features predict proprioceptive state transitions: per-stream zero-sum temporal kernels (initialized as first-order differences) extract motion codes, a cycle-consistent head predicts proprioceptive innovations forward and backward, and auxiliary heads regress end-effector-to-target alignment and binary engagement; masked cross-modal reconstruction with stop-gradient targets adds robustness. The frozen encoder feeds an asymmetric actor–critic PPO policy trained in Isaac Lab with domain randomization and deployed on a Franka FR3 without real-world fine-tuning or pose estimation. Across four tasks (PegInsert, GearMesh, JointAssemble, KnobTighten), PACE reports 93.3% average hardware success with a 2.7-percentage-point average sim-to-real drop, versus 77.5%/18.9 pp for pose-based PPO and 68.3%/18.4 pp for AugInsert; removing anchoring raises the drop to 25.5 pp while simulation success falls only 4.7 pp. Hardware evaluation uses 30 trials per method–task pair with best-simulation checkpoint selection; Wilson intervals (8–17 pp half-width) exceed the headline gap. Limitations: single robot, quasi-static tasks, untested seed variability on hardware, and no comparison to the closest multisensory-pretraining prior (MSDP).