HFHuman: High-Fidelity Human Reconstruction From Single Image With Multi-Modality Fusion

Area of Science:

• Computer Vision
• 3D Reconstruction
• Computer Graphics

Background:

• Accurate 3D human model reconstruction from single images is crucial for virtual reality (VR) applications.
• Existing methods struggle with occlusions and viewpoint limitations, often relying on 3D features that are difficult to predict from monocular images.
• This difficulty hinders effective fusion of 2D and 3D information.

Purpose of the Study:

• To introduce HFHuman, a novel approach for high-fidelity human reconstruction from a single image.
• To address the challenges of fusing 2D and 3D information for more accurate and realistic human models.
• To improve geometric priors by directly fusing multi-modal image data.

Main Methods:

• HFHuman employs a multi-modality fusion strategy, integrating geometric and depth information directly from images.
• Key innovations include a parallel reconstruction framework for simultaneous whole-body geometry and depth, a pixel-voxel feature fusion strategy, and a depth-refinement technique using RGB, surface normals, and depth maps.
• A parameterized 3D human model is refined under progressive depth guidance.

Main Results:

• HFHuman achieves more accurate and realistic human reconstructions compared to existing methods.
• The approach effectively fuses multiple modalities, overcoming limitations in predicting 3D features from monocular images.
• Experimental results show HFHuman outperforms state-of-the-art methods in 3D human body reconstruction.

Conclusions:

• HFHuman sets a new standard for high-fidelity 3D human reconstruction from single images.
• The multi-modality fusion approach successfully integrates 2D and 3D information for improved results.
• The method offers a significant advancement for virtual reality and related applications.