NeRFFaceShop: Learning a Photo-Realistic 3D-Aware Generative Model of Animatable and Relightable Heads From...

Area of Science:

• Computer Vision and Graphics
• 3D Generative Models
• Neural Rendering

Background:

• Generating animatable and relightable 3D faces is crucial for computer vision and graphics applications.
• Existing methods often struggle to integrate animation and relighting due to computational costs and representation incompatibilities.
• Explicit shading models face training challenges for realistic relighting, while implicit lighting struggles with deforming spatial points in animation.

Purpose of the Study:

• To develop a disentangled generative model for 3D animatable and relightable heads.
• To enable realistic control over both facial motion and lighting conditions simultaneously.
• To overcome limitations of previous methods by proposing a compatible animation representation and a lightweight lighting representation.

Main Methods:

• Introduced a lightweight and effective lighting representation.
• Developed a compatible animation representation for disentangled 3D head generation.
• Trained the model on large-scale, in-the-wild videos, avoiding reliance on morphable models.
• Utilized domain transfer for applications lacking video data.

Main Results:

• Synthesized geometrically consistent and detailed 3D head motion.
• Achieved disentangled control over lighting conditions.
• Demonstrated compatibility with morphable models for avatar driving.
• Showcased broader applicability through domain transfer.

Conclusions:

• The proposed method successfully achieves disentangled 3D animatable and relightable head generation.
• The approach offers realistic control over animation and relighting, learning from diverse video data.
• The method is versatile, compatible with existing models, and extendable to new domains.