NeuralFeels with neural fields: Visuotactile perception for in-hand manipulation

Area of Science:

• Robotics
• Artificial Intelligence
• Computer Vision
• Sensor Fusion

Background:

• Human-level robotic dexterity requires sophisticated spatial awareness, particularly for in-hand manipulation of novel objects.
• Current in-hand perception systems predominantly rely on vision, limiting their effectiveness with occluded objects and a priori unknown object models.
• Visual occlusion is a significant challenge in robotic manipulation, hindering the development of robust in-hand manipulation capabilities.

Purpose of the Study:

• To develop a multimodal sensing approach combining vision and touch for estimating object pose and shape during in-hand manipulation.
• To overcome the limitations of vision-only systems, especially in scenarios with significant visual occlusion.
• To advance robot dexterity by creating a robust perception backbone for in-hand object interaction.

Main Methods:

• Introduced NeuralFeels, a method that encodes object geometry using an online learned neural field.
• Jointly tracked object pose and shape by optimizing a pose graph problem, integrating vision and touch data.
• Utilized a proprioception-driven policy for interaction with objects in both simulation and real-world experiments.

Main Results:

• Achieved a final reconstruction F-score of 81% for object geometry.
• Demonstrated an average pose drift of 4.7 millimeters, reduced to 2.3 millimeters with known object models.
• Showcased significant improvements in tracking under heavy visual occlusion (up to 94% compared to vision-only methods).

Conclusions:

• Multimodal sensing, integrating touch with vision, refines and disambiguates object perception during in-hand manipulation.
• The NeuralFeels method provides a robust perception backbone for advancing robot dexterity, particularly in challenging occlusion scenarios.
• The release of the FeelSight dataset aims to facilitate benchmarking and further research in multimodal in-hand perception.