GSwap: Realistic Head Swapping With Dynamic Neural Gaussian Field

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Existing video head-swapping methods often struggle with 3D consistency, natural expression synthesis, and seamless background integration.
• Previous approaches using 2D generative models or 3D Morphable Face Models (3DMM) have inherent limitations in holistic head modeling and realism.

Purpose of the Study:

• To develop a novel and realistic video head-swapping system that addresses the limitations of current state-of-the-art methods.
• To achieve high-fidelity, 3D-consistent portrait rendering with natural head-torso relationships and seamless motion dynamics.

Main Methods:

• Introduced GSwap, a system leveraging dynamic neural Gaussian portrait priors and a 3D Gaussian feature field within a full-body SMPL-X surface.
• Adapted a pretrained 2D portrait generative model for efficient domain adaptation using few-shot learning.
• Proposed a neural re-rendering strategy for seamless foreground-background integration, eliminating blending artifacts.

Main Results:

• GSwap significantly advances the state of the art in face and head replacement, demonstrating superior performance over existing methods.
• Achieved high-fidelity, 3D-consistent portrait rendering with natural facial expressions and seamless motion.
• Successfully integrated synthesized foregrounds with original backgrounds, enhancing overall realism and eliminating artifacts.

Conclusions:

• GSwap offers a robust solution for video head-swapping, outperforming previous techniques in visual quality, temporal coherence, identity preservation, and 3D consistency.
• The dynamic neural Gaussian field approach combined with neural re-rendering provides a significant leap forward in realistic and consistent digital human synthesis.