Bilateral Cross-Modal Fusion Network for Robot Grasp Detection.

Area of Science:

• Robotics
• Computer Vision
• Artificial Intelligence

Background:

• Accurate target pose estimation using RGB and depth data is crucial for vision-based robot grasping.
• Existing methods face challenges in effectively fusing multimodal information for precise grasp detection.

Purpose of the Study:

• To propose a novel tri-stream cross-modal fusion architecture for 2-DoF visual grasp detection.
• To enhance the aggregation of multiscale information from RGB and depth data.
• To improve the accuracy and robustness of robot grasping systems.

Main Methods:

• Developed a tri-stream cross-modal fusion architecture integrating RGB and depth information.
• Introduced a novel modal interaction module (MIM) utilizing spatial-wise cross-attention.
• Employed channel interaction modules (CIM) for enhanced cross-modal feature aggregation.
• Utilized a hierarchical structure with skipping connections for multiscale information aggregation.

Main Results:

• Achieved 99.4% image-wise detection accuracy on the Cornell dataset and 96.7% on the Jacquard dataset.
• Reached 97.8% object-wise detection accuracy on the Cornell dataset and 94.6% on the Jacquard dataset.
• Demonstrated a 94.5% success rate in physical experiments using a 6-DoF Elite robot.

Conclusions:

• The proposed tri-stream cross-modal fusion architecture significantly improves visual grasp detection accuracy.
• The novel MIM and CIM modules effectively capture and aggregate cross-modal and multiscale features.
• The method shows superior performance in both dataset evaluations and real-world robotic grasping tasks.