Sim-to-Real for High-Resolution Optical Tactile Sensing: From Images to Three-Dimensional Contact Force...

Area of Science:

• Robotics
• Computer Vision
• Materials Science

Background:

• Vision-based tactile sensors capture high-resolution tactile field information, like contact force distribution.
• Extracting this information is challenging, often requiring extensive real-world training data for learning-based methods.

Purpose of the Study:

• To develop a simulation strategy for generating realistic tactile images for vision-based tactile sensors.
• To train artificial neural networks using entirely synthetic data, bypassing the need for real-world data collection.

Main Methods:

• Simulating material deformation in a finite element environment under various contact conditions.
• Projecting internal spherical particle motion to generate simulated tactile images.
• Mapping image features to 3D contact force distributions using finite-element simulations for ground truth.

Main Results:

• An artificial neural network trained solely on synthetic data achieved high accuracy on real-world tactile images.
• The trained model demonstrated transferability across multiple tactile sensors without retraining.
• The approach is suitable for efficient real-time inference.

Conclusions:

• Simulation-based generation of tactile images is a viable alternative to real-world data collection.
• Synthetic data training enables accurate, transferable, and efficient vision-based tactile sensing models.
• This method significantly reduces the data acquisition burden in tactile sensing research.