PA-Tran: Learning to Estimate 3D Hand Pose with Partial Annotation.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• 3D Hand Pose Estimation (HPE) from single RGB images is challenging due to occlusions.
• Existing HPE methods often overlook partially visible keypoints, limiting their real-world applicability.
• Handling partial annotations in training data is crucial for robust HPE models.

Purpose of the Study:

• To develop a novel deep-learning framework (PA-Tran) for 3D HPE from single RGB images with partial annotations.
• To jointly estimate keypoint status (observed/unobserved) and 3D hand pose.
• To investigate the efficiency of training with partial annotations and introduce new benchmark datasets.

Main Methods:

• Proposed PA-Tran framework with two dependent branches: a Transformer encoder for regression and a CNN for classification.
• Introduced Selective Mask Training (SMT) objective using binary encoding for keypoint status.
• Developed two new datasets: APDM-Hand (synthetic) and PD-APDM-Hand (real-world, Parkinson's Disease patients with partial annotations).

Main Results:

• PA-Tran achieves higher estimation accuracy on both new and general datasets.
• Training with partial annotations (around 85%) is more efficient, yielding the best performance (6.0 PA-MPJPE).
• The framework effectively handles partially visible keypoints and partial annotation availability.

Conclusions:

• PA-Tran offers a robust solution for 3D HPE from single RGB images, particularly in scenarios with occlusions and partial annotations.
• The SMT objective and joint estimation of pose and status are key to the framework's success.
• The new datasets facilitate further research in challenging 3D HPE applications.