Bio-Inspired Proprioceptive Touch of a Soft Finger with Inner-Finger Kinesthetic Perception

Area of Science:

• Robotics
• Computer Vision
• Biomimetics

Background:

• In-hand object pose estimation is crucial for robotic manipulation but is hindered by occlusion from the hand and object.
• Existing methods often struggle with the complex interactions and deformations involved in grasping.

Purpose of the Study:

• To develop a soft finger with integrated inner vision and kinesthetic sensing for robust in-hand object pose estimation.
• To create an end-to-end framework capable of estimating object pose and category from raw sensor data.

Main Methods:

• A soft finger with a flexible skeleton and adaptive skin was designed, incorporating an internal camera for visual feedback.
• Skeleton deformations during object interaction were captured and processed using an encoder for kinesthetic information.
• An end-to-end neural network processed raw images and kinesthetic data for simultaneous pose and category estimation.

Main Results:

• The proposed framework achieved an impressive average pose error of 2.02 mm and 11.34 degrees across seven test objects.
• Object classification accuracy reached 99.05%, demonstrating the system's effectiveness.
• The integrated approach successfully addressed occlusion challenges in in-hand object manipulation.

Conclusions:

• The soft finger with inner vision and kinesthetic sensing offers a promising solution for accurate in-hand object pose estimation.
• This biomimetic approach enhances robotic grasping and manipulation by providing rich, multi-modal sensory feedback.
• The developed end-to-end framework demonstrates high performance in both pose estimation and object recognition.