CGOF++: Controllable 3D Face Synthesis With Conditional Generative Occupancy Fields

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Existing 2D controllable face synthesis methods struggle with consistency under significant pose and expression variations.
• Recent advances in image generation models enable high-fidelity face synthesis with some controllability.

Purpose of the Study:

• To propose a novel Neural Radiance Field (NeRF)-based framework for controllable 3D face synthesis.
• To enhance 3D controllability by integrating explicit 3D conditions from 3D face priors.

Main Methods:

• Developed a conditional Generative Occupancy Field (cGOF++) built upon the EG3D generative model.
• Incorporated a 3D Morphable Model (3DMM) mesh to enforce face shape conformity.
• Integrated 3D landmark loss and volume warping loss for fine-grained 3D shape control.

Main Results:

• The proposed framework generates high-fidelity 3D face images.
• Achieved more precise 3D controllability compared to state-of-the-art 2D-based methods.
• Demonstrated improved consistency in generated face images under large expression and pose changes.

Conclusions:

• The NeRF-based conditional 3D face synthesis framework offers superior 3D controllability.
• The method effectively addresses the limitations of 2D generative models for face synthesis.
• The framework enables precise control over fine-grained 3D face shapes.