Diffusion Models are Efficient Data Generators for Human Mesh Recovery.

Area of Science:

• Computer Vision
• Machine Learning
• 3D Human Pose and Shape Estimation

Background:

• Current 3D human pose and shape (HPS) methods rely on indoor motion capture or computer graphics (CG) data, lacking real-world diversity.
• Existing datasets fail to capture authentic human identities and in-the-wild background scenes, hindering generalization.

Purpose of the Study:

• To develop a novel data generation pipeline, HumanWild, for creating synthetic 3D human images with accurate mesh annotations.
• To improve the generalization performance of 3D HPS methods on diverse, real-world scenes by leveraging generative models.

Main Methods:

• Utilized diffusion models and a customized ControlNet for generating human images and 3D mesh annotations.
• Employed a 3D parametric model (SMPL-X) for precise 2D keypoint, depth map, and surface normal generation.
• Incorporated foundation segmentation and 2D vertex estimation for data filtering and label rectification.

Main Results:

• Successfully generated large-scale, in-the-wild human images with high-quality 3D mesh annotations.
• Demonstrated the effectiveness of synthetic data through comprehensive ablation studies on HPS benchmarks.
• Showcased the pipeline's flexibility for tasks like human interaction and wide-angle captures.

Conclusions:

• Generative models offer a powerful, complementary approach to CG data for 3D HPS.
• The HumanWild pipeline effectively scales data generation for in-the-wild 3D human recovery.
• This work facilitates more robust and versatile 3D human pose and shape estimation in real-world scenarios.