ADG-Net: A Sim2Real Multimodal Learning Framework for Adaptive Dexterous Grasping.

Area of Science:

• Robotics
• Machine Learning
• Computer Vision

Background:

• Dexterous grasping is crucial for robotic manipulation.
• Bridging the gap between simulation and real-world (sim2real) performance remains a challenge.
• Integrating multimodal sensory data can enhance grasp robustness.

Purpose of the Study:

• To propose a novel simulation-to-real (sim2real) multimodal learning framework for adaptive dexterous grasping and grasp status prediction.
• To develop an adaptive dexterous grasping neural network (ADG-Net) capable of learning grasp principles and predicting grasp parameters.
• To validate the framework's effectiveness in both simulated and physical environments.

Main Methods:

• A two-stage approach using Isaac Gym and pluggable modules to simulate dexterous grasps with multimodal sensing data (RGB-D, joint angles, tactile forces).
• Collection of over 500,000 multimodal synthetic grasping scenarios for neural network training.
• Training of ADG-Net, incorporating an attention mechanism and graph convolutional neural network (GCNN) for multimodal information fusion.

Main Results:

• ADG-Net successfully detects feasible grasp parameters from RGB-D images and optimizes them using multimodal data.
• Achieved an average success rate of 92% for grasping isolated unseen objects.
• Achieved an average success rate of 83% for grasping stacked objects in physical experiments.

Conclusions:

• The proposed adaptive dexterous grasping method significantly outperforms state-of-the-art methods.
• The sim2real multimodal learning framework demonstrates robust performance in complex grasping scenarios.
• The developed ADG-Net offers a promising solution for advanced robotic grasping capabilities.