CrossFuNet: RGB and Depth Cross-Fusion Network for Hand Pose Estimation

Area of Science:

• Computer Vision
• Machine Learning
• Robotics

Background:

• RGB-based 3D hand pose estimation faces challenges like self-occlusion and depth ambiguity.
• Depth-based methods are limited by distance sensitivity and indoor-only applicability.
• Combining RGB and depth data offers a promising approach to mitigate individual modality weaknesses.

Purpose of the Study:

• To develop an effective fusion network for enhanced 3D hand pose estimation.
• To address the limitations of existing RGB and depth-based hand pose estimation techniques.
• To introduce a novel fusion module for integrating multi-modal data.

Main Methods:

• A novel RGB and depth information fusion network, CrossFuNet, is proposed.
• Input RGB images and depth maps are processed by separate subnetworks.
• A unique fusion module combines features from both modalities before regressing 3D key-points via heatmaps.

Main Results:

• The CrossFuNet model was validated on two public datasets.
• Experimental results demonstrate superior performance compared to state-of-the-art methods.
• The proposed fusion approach effectively improves the accuracy of 3D hand pose estimation.

Conclusions:

• The CrossFuNet model successfully integrates RGB and depth information for robust 3D hand pose estimation.
• The novel fusion strategy significantly enhances accuracy, overcoming limitations of single-modality approaches.
• This work advances the field of 3D hand pose estimation with a more practical and accurate solution.