Scene flow based deep network for hand reconstruction using depth images

Area of Science:

• Computer Vision
• 3D Reconstruction
• Machine Learning

Background:

• Accurate 3D hand reconstruction is a challenging computer vision problem.
• Existing methods often neglect temporal information, limiting stable hand tracking.
• Multi-view depth imaging offers rich data for hand pose estimation.

Purpose of the Study:

• To develop a novel pipeline, HandFlowNet, for accurate 3D hand reconstruction from consecutive multi-view depth images.
• To incorporate temporal information for improved stability in hand tracking.
• To achieve state-of-the-art performance on benchmark datasets.

Main Methods:

• Converting multi-view depth images into a single point cloud.
• Estimating scene flow of hand mesh vertices to deduce temporal information between frames.
• Utilizing a graph convolutional network for refining hand mesh vertices with local and global features.

Main Results:

• HandFlowNet successfully deduces temporal information from sequential depth frames.
• The scene flow is applied as an offset for accurate vertex estimation.
• The graph convolutional network refines mesh vertices for enhanced accuracy.
• State-of-the-art performance achieved on DexYCB and HO3D benchmarks.

Conclusions:

• HandFlowNet provides a robust pipeline for 3D hand reconstruction.
• The integration of temporal information significantly enhances hand tracking stability.
• The proposed method sets a new benchmark for accuracy in real-world hand pose estimation.