3D-CariGAN: An End-to-End Solution to 3D Caricature Generation From Normal Face Photos

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Existing 3D caricature generation methods require specific inputs like sketches, which are challenging for non-professionals.
• The significant domain gap between normal 2D face photos and 3D exaggerated caricatures presents a major technical hurdle.

Purpose of the Study:

• To develop an end-to-end deep neural network model for generating high-quality 3D caricatures directly from standard 2D face photographs.
• To bridge the domain gap between 2D facial photos and 3D caricature representations.

Main Methods:

• Construction of a large dataset (5,343 meshes) of 3D caricatures to establish a Principal Component Analysis (PCA) model.
• Reconstruction of a 3D head model from the input 2D photo and utilizing its PCA representation for shape correspondence.
• Introduction of novel character and caricature loss functions informed by psychological studies on caricature perception.

Main Results:

• The proposed deep neural network successfully generates high-quality 3D caricatures from normal 2D face photos.
• The PCA model and novel loss functions effectively address the domain gap between source and target data.
• A two-level user study validated the system's capability in producing realistic and high-quality 3D caricatures.

Conclusions:

• The developed system offers a user-friendly solution for 3D caricature generation, directly from 2D images.
• This research advances the field of 3D face modeling and artistic style transfer.
• The method demonstrates the potential of deep learning in creating stylized 3D content from everyday photographs.