HandBooster+: Boosting 3D Hand-Mesh Reconstruction From Data Synthesis to Progressive Multi-Hypothesis Aggregation.

Area of Science:

• Computer Vision
• Machine Learning
• 3D Reconstruction

Background:

• Reconstructing 3D hand meshes from single images is challenging due to limited dataset diversity and occluded regions.
• Existing data synthesis methods face a 'syn-to-real' gap, and probabilistic approaches often require ground truth for hypothesis selection.
• Deterministic reconstruction methods struggle with ambiguity in occluded hand areas.

Purpose of the Study:

• To develop a novel approach using diffusion models for robust 3D hand mesh reconstruction.
• To address data diversity limitations and ambiguity in occluded hand regions.
• To improve the accuracy and reliability of 3D hand mesh generation from single images.

Main Methods:

• Introduced HandBooster for conditional synthesis and sampling to generate diverse, realistic data with 3D annotations.
• Developed HandBooster+, a probabilistic diffusion model employing progressive multi-hypothesis aggregation.
• Leveraged diffusion models for both realistic data generation and probabilistic reconstruction.

Main Results:

• The proposed methods significantly improve upon existing baselines for 3D hand mesh reconstruction.
• Achieved state-of-the-art (SOTA) performance on the HO3D and DexYCB benchmarks.
• Demonstrated effective handling of data diversity and occluded hand regions.

Conclusions:

• Diffusion models offer a powerful framework for addressing key challenges in 3D hand mesh reconstruction.
• HandBooster and HandBooster+ provide a robust and effective solution for generating accurate 3D hand meshes from single images.
• The approach enhances data realism and improves reconstruction accuracy, particularly in complex scenarios.