Learning a simulation-based visual policy for real-world peg in unseen holes

Area of Science:

• Robotics
• Computer Vision
• Machine Learning

Background:

• Robotic manipulation, particularly peg-in-hole tasks, requires precise control and adaptability to variations.
• Current methods often struggle with generalization to unseen shapes and require extensive real-world training data.
• Bridging the gap between simulation and real-world performance (sim-to-real transfer) remains a significant challenge in robotics.

Purpose of the Study:

• To develop a learning-based visual peg-in-hole system capable of adapting to arbitrary unseen shapes with minimal sim-to-real cost.
• To decouple the generalization of the sensory-motor policy from perception and control modules for improved adaptability.
• To enable efficient training and rapid deployment in real-world scenarios.

Main Methods:

• A framework comprising a segmentation network (SN), virtual sensor network (VSN), and controller network (CN) was proposed.
• The VSN was trained to measure object pose from segmented images, enabling shape-agnostic pose measurement.
• A generic peg-in-hole policy was trained using the CN, with only the SN fine-tuned for real-world transfer, minimizing adaptation cost.

Main Results:

• The system demonstrated successful adaptation to unseen shapes with minimal sim-to-real transfer cost.
• A peg-in-hole task was achieved with a 10/10 success rate in 2-3 seconds using an electric vehicle charging system.
• The approach required only hundreds of automatically labeled samples for SN transfer, significantly reducing data collection efforts.

Conclusions:

• The proposed learning-based visual peg-in-hole framework effectively addresses the sim-to-real transfer problem for robotic manipulation tasks.
• Decoupling perception and control modules allows for fast adaptation to novel shapes with minimal fine-tuning.
• The system shows practical viability for complex manipulation tasks, as evidenced by its high success rate in automated EV charging.