Graph-Based and Multi-Stage Constraints for Hand-Object Reconstruction

Area of Science:

• Computer Vision
• Robotics
• 3D Reconstruction

Background:

• Reconstructing hand-object interactions from single views is difficult due to occlusion and physical plausibility requirements.
• Existing methods often process hands and objects separately or use late-stage post-processing for constraints.

Purpose of the Study:

• To develop a novel framework for holistic hand-object interaction reconstruction.
• To progressively enforce physical consistency and geometric accuracy throughout the reconstruction pipeline.

Main Methods:

• Input: RGB-D image.
• Adaptive feature fusion for robust color and depth integration.
• Structural priors and texture cues for refined 3D pose initialization.
• Dense mutual attention for sparse-to-dense mesh recovery and interaction dependency capture.
• Signed Distance Function (SDF) representation for contact surfaces to ensure non-penetration.

Main Results:

• Significant improvements on ObMan and DexYCB benchmarks, outperforming state-of-the-art.
• ObMan dataset: hand CDh 0.077 cm², object CDo 0.483 cm².
• DexYCB dataset: hand CDh 0.251 cm², object CDo 1.127 cm².

Conclusions:

• The proposed holistic, multi-stage collaborative optimization framework effectively reconstructs hand-object interactions.
• The method achieves high physical plausibility and geometric accuracy, surpassing existing techniques.