A Recognition Method for Soft Objects Based on the Fusion of Vision and Haptics

Area of Science:

• Robotics
• Computer Vision
• Artificial Intelligence

Background:

• Object recognition often requires integrating multiple sensing modalities due to limitations of single methods.
• Vision is a primary sense but struggles with challenges like low light or visually similar objects with different internal properties.
• Haptic sensing provides crucial contact and physical information complementary to vision.

Purpose of the Study:

• To develop an end-to-end method for robust object perception by fusing visual and haptic information.
• To improve the accuracy of object recognition, particularly for soft objects with subtle differences.
• To explore the utility of extracted physical features for subsequent manipulation tasks.

Main Methods:

• Utilized the YOLO deep network for visual feature extraction.
• Employed haptic explorations for haptic feature extraction.
• Integrated visual and haptic features using a graph convolutional network.
• Object recognition performed using a multi-layer perceptron.

Main Results:

• The visual-haptic fusion method demonstrated superior performance in distinguishing soft objects with varied interior fillers compared to vision-only or other methods.
• Achieved an average recognition accuracy of 0.95, a significant improvement from the vision-only mean average precision (mAP) of 0.502.
• Demonstrated the potential of extracted physical features for soft object manipulation.

Conclusions:

• Visual-haptic fusion is an effective strategy for enhancing object perception robustness.
• The proposed method offers a significant advancement in recognizing visually similar objects with differing internal characteristics.
• The integrated approach provides valuable physical insights for robotic manipulation.