3D hand pose and mesh estimation via a generic Topology-aware Transformer model

Area of Science:

• Computer Vision
• Robotics
• Machine Learning

Background:

• Accurate 3D hand pose and mesh estimation are crucial for Human-Robot Interaction (HRI).
• Severe self-occlusion and high self-similarity in hand data present significant challenges for existing methods.
• Existing approaches struggle to effectively resolve ambiguities caused by invisible or similar hand joints.

Purpose of the Study:

• To propose a novel network, HandGCNFormer, for improved 3D hand pose estimation from depth images.
• To incorporate prior knowledge of hand kinematic topology into a Transformer network.
• To enhance the modeling of long-range contextual information and local topological connections for greater accuracy.

Main Methods:

• Developed HandGCNFormer, a Topology-aware Transformer network utilizing depth image input.
• Introduced a novel Graphformer decoder with an additional Node-offset Graph Convolutional layer (NoffGConv).
• Implemented a Topology-aware head, replacing standard MLP, to leverage local topological constraints.

Main Results:

• Achieved state-of-the-art performance in 3D hand pose estimation across four challenging datasets (Hands2017, NYU, ICVL, MSRA).
• Extended the framework to HandGCNFormer-Mesh for 3D hand mesh estimation, producing Mano parameters.
• Demonstrated competitive results on the HO-3D dataset, particularly in handling occlusions.

Conclusions:

• The proposed HandGCNFormer effectively addresses challenges in 3D hand pose estimation by integrating topological priors and advanced network architectures.
• The novel Graphformer decoder and Topology-aware head significantly improve accuracy and robustness.
• The extended HandGCNFormer-Mesh shows promise for 3D hand mesh estimation in complex, occluded scenarios.