MuNeRF: Robust Makeup Transfer in Neural Radiance Fields.

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Facial makeup transfer is crucial for fashion e-commerce and virtual avatars.
• Existing methods, often using generative adversarial networks, lack consistency across varying poses and expressions.
• Maintaining makeup integrity under different facial geometries is a significant challenge.

Purpose of the Study:

• To develop a robust facial makeup transfer method ensuring consistent style application across diverse poses and expressions.
• To leverage implicit 3D representations for geometric and appearance consistency in makeup transfer.
• To improve the visual quality and faithfulness of synthesized makeup effects.

Main Methods:

• Utilized neural radiance fields (NeRFs) for implicit 3D reconstruction of facial geometry.
• Developed a two-stage approach: a basic NeRF module for geometry and a makeup module for style transfer.
• Introduced a novel hybrid makeup loss and a patch-based discriminator in UV texture space for enhanced control and alignment.

Main Results:

• Achieved consistent makeup transfer across various poses and expressions, overcoming limitations of previous methods.
• The hybrid makeup loss significantly improved visual quality and faithfulness of the transferred makeup.
• The patch-based discriminator in UV space provided accurate control over synthesized makeup distribution.

Conclusions:

• The proposed NeRF-based method offers superior robustness and consistency in facial makeup transfer.
• The novel loss function and discriminator contribute to high-fidelity and controllable makeup synthesis.
• This approach advances the state-of-the-art for virtual makeup applications in e-commerce and avatar generation.