Generating High-Resolution 3D Faces and Bodies Using VQ-VAE-2 with PixelSNAIL Networks on 2D Representations

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• 3D human shape modeling is crucial for industries like healthcare, apparel, and entertainment.
• Existing methods for 3D shape synthesis face challenges in generating realistic and diverse outputs.

Purpose of the Study:

• To develop a novel method for synthesizing 3D human faces and bodies.
• To represent complex 3D geometries using 2D image-based techniques for improved synthesis.

Main Methods:

• A non-bijective 3D-to-2D conversion method was proposed, representing 3D body meshes as multiple 2D projections.
• A vector-quantized variational autoencoder (VQ-VAE-2) was trained on 2D representations to learn latent features.
• A PixelSNAIL autoregressive model was employed to generate novel synthetic 3D shapes from learned latent representations.

Main Results:

• The proposed method successfully models 3D faces on a 2D grid, generating synthetic faces statistically closer to real ones than PCA-based methods.
• Quantitative evaluation using specificity and diversity metrics demonstrated superior performance for synthetic faces.
• Initial experiments on 3D body geometry show promising results, though further research is needed to match test set statistics.

Conclusions:

• The 2D image-based approach offers an effective strategy for 3D human shape synthesis, particularly for faces.
• The novel 3D-to-2D conversion and VQ-VAE-2 framework provide a robust foundation for generating realistic synthetic 3D human models.
• Future work will focus on refining the body modeling aspect to achieve comparable results to face synthesis.