Consistent 3D Hand Reconstruction in Video via Self-Supervised Learning.

Area of Science:

• Computer Vision
• 3D Reconstruction
• Machine Learning

Background:

• Accurate 3D hand reconstruction is crucial for applications like augmented reality and robotics.
• Existing methods often require extensive 3D hand annotations, which are costly and time-consuming to acquire.
• Leveraging easily obtainable 2D keypoints and image texture can mitigate the need for 3D supervision.

Purpose of the Study:

• To propose S²HAND, a novel self-supervised model for accurate and consistent 3D hand reconstruction from monocular RGB video.
• To demonstrate that 2D keypoints and image texture are sufficient for effective 3D hand geometry and texture estimation.
• To develop S²HAND(V) by incorporating temporal consistency constraints for enhanced performance on video data.

Main Methods:

• Developed S²HAND, a self-supervised model that jointly estimates 3D hand pose, shape, texture, and camera viewpoint from single RGB images.
• Utilized detected 2D hand keypoints as the primary supervision signal, reducing dependency on 3D ground truth.
• Extended S²HAND to S²HAND(V) by leveraging unlabeled video data and enforcing motion, texture, and shape consistency across frames.

Main Results:

• S²HAND achieves comparable performance to fully-supervised methods in single-frame reconstruction.
• S²HAND(V) significantly improves 3D hand reconstruction accuracy and consistency by utilizing temporal information from videos.
• The model effectively reconstructs accurate 3D hand geometry and texture without requiring 3D annotations.

Conclusions:

• Self-supervised learning is a viable and effective approach for 3D hand reconstruction.
• The proposed S²HAND and S²HAND(V) models offer a powerful alternative to supervised methods, especially in data-scarce scenarios.
• The method demonstrates the potential of using readily available 2D cues for complex 3D scene understanding.