OccFusion: Rendering Occluded Humans with Generative Diffusion Priors.

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Existing human rendering methods often assume complete visibility, which is unrealistic.
• Real-world scenarios frequently involve occlusions, leading to partial human visibility in input data.

Purpose of the Study:

• To develop an efficient and high-fidelity human rendering approach for scenarios with occlusions.
• To enable complete human geometry reconstruction from partially visible inputs.

Main Methods:

• OccFusion utilizes efficient 3D Gaussian splatting supervised by pretrained 2D diffusion models.
• A three-stage pipeline involves mask generation, 3D Gaussian optimization with Score-Distillation Sampling (SDS), and in-context inpainting for refinement.

Main Results:

• OccFusion successfully generates complete human geometry from partial visibility masks.
• The method achieves state-of-the-art performance in rendering occluded humans on benchmark datasets like ZJU-MoCap and OcMotion.

Conclusions:

• OccFusion provides an effective solution for high-fidelity human rendering in the presence of occlusions.
• The proposed pipeline demonstrates the potential of combining 3D Gaussian splatting and diffusion models for challenging computer vision tasks.