HandRT: Simultaneous Hand Shape and Appearance Reconstruction With Pose Tracking From Monocular RGB-D Video

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Personalized 3D hand avatar reconstruction is crucial for applications like virtual reality and augmented reality.
• Existing methods often require known initial hand poses or struggle with arbitrary poses and illumination variations.

Purpose of the Study:

• To develop a robust method for reconstructing personalized 3D hand avatars from monocular RGB-D video.
• To enable accurate reconstruction of hand shape, appearance, pose, and lighting without prior pose knowledge.

Main Methods:

• HandRT utilizes a differentiable rendering framework with Monte Carlo path tracing to jointly optimize hand parameters.
• An articulated registration energy based on iterative closest point (ICP) is introduced for pose tracking across frames.
• The method incorporates a physically-based shading model to handle unknown illumination conditions.

Main Results:

• HandRT successfully reconstructs personalized hand avatars from arbitrary poses with high accuracy, outperforming existing methods.
• The method achieves precise tracking of the reconstructed avatar from both RGB and RGB-D inputs.
• Evaluations show significantly more accurate mesh recovery compared to state-of-the-art techniques on public and custom datasets.

Conclusions:

• HandRT offers a significant advancement in personalized 3D hand avatar reconstruction from monocular RGB-D video.
• The method's ability to handle unknown poses and illumination makes it highly versatile for real-world applications.
• The proposed approach sets a new benchmark for accuracy in hand mesh recovery and avatar generation.