Cross-Domain and Disentangled Face Manipulation With 3D Guidance

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• 3D Morphable Model (3DMM) based face manipulation is effective for photorealistic faces.
• Existing methods struggle with out-of-domain faces like paintings or cartoons due to domain-specific model requirements.

Purpose of the Study:

• To develop a universal face manipulation method applicable to arbitrary domains using a single human 3DMM.
• To enable semantic control over diverse facial attributes in out-of-domain face images.

Main Methods:

• A two-step approach: 1) Disentangled mapping from 3DMM parameters to a pre-trained StyleGAN2 latent space. 2) Cross-domain adaptation to bridge domain discrepancies.
• Enforcing a consistent latent space embedding for seamless adaptation.

Main Results:

• High-quality semantic manipulation achieved across various face domains (paintings, cartoons, etc.).
• Controllability demonstrated for major 3D facial attributes: pose, expression, shape, albedo, and illumination.
• An intuitive editing interface was developed for user-friendly control and real-time feedback.

Conclusions:

• The proposed method is the first to enable manipulation of faces in arbitrary domains using a human 3DMM.
• The approach offers superior performance and broad applicability for 3D face image manipulation.