FSGANv2: Improved Subject Agnostic Face Swapping and Reenactment.

Area of Science:

• Computer Vision
• Deep Learning
• Computer Graphics

Background:

• Existing face swapping and reenactment methods often require subject-specific training data.
• Accurate manipulation of facial pose and expression, especially in video sequences, remains a challenge.
• Seamlessly blending swapped faces while preserving original skin tone and lighting conditions is difficult.

Purpose of the Study:

• To introduce a subject-agnostic face swapping and reenactment framework (FSGAN).
• To develop a novel deep learning approach for realistic face reenactment, handling pose and expression variations.
• To achieve seamless face blending with preserved identity, skin color, and lighting.

Main Methods:

• A novel iterative deep learning approach for face reenactment applicable to single images or video sequences.
• Continuous interpolation of face views using Delaunay Triangulation and barycentric coordinates for video sequences.
• A face completion network for occluded regions and a face blending network employing a novel Poisson blending loss (Poisson optimization + perceptual loss).

Main Results:

• Demonstrated a subject-agnostic face swapping scheme, eliminating the need for training on specific subjects.
• Achieved significant adjustments for pose and expression variations in face reenactment.
• Outperformed existing state-of-the-art systems both qualitatively and quantitatively in face swapping and reenactment tasks.

Conclusions:

• The proposed Face Swapping GAN (FSGAN) offers a robust and versatile solution for face swapping and reenactment.
• The subject-agnostic approach and advanced blending techniques lead to superior and more natural results.
• FSGAN advances the state-of-the-art in generative adversarial networks for facial image manipulation.