SwapController: Toward Improving Identity and Attribute Control for Diffusion-Based Face Swapping

Area of Science:

• Computer Vision
• Artificial Intelligence
• Deep Learning

Background:

• Diffusion models offer high-fidelity face generation but struggle with identity and attribute consistency.
• Accurate modeling of facial representations remains a challenge in current face-swapping technologies.

Purpose of the Study:

• To introduce SwapController, a novel diffusion-based framework for high-fidelity face swapping.
• To improve identity and attribute consistency in generated faces using advanced modeling techniques.

Main Methods:

• Utilizing 3D facial priors for explicit structure supervision and accurate head structure control.
• Developing multi-grained source identity embeddings for enhanced identity preservation.
• Creating identity-shifted attribute embeddings for improved attribute modeling.
• Implementing a timestep-aware identity optimization objective for diffusion models.

Main Results:

• SwapController demonstrates effectiveness in generating identity-consistent portraits while preserving target attributes.
• Achieved a 98.32 ID Retrieval score, surpassing the state-of-the-art by 7.32.
• The framework successfully balances high-fidelity generation with precise control over facial identity and attributes.

Conclusions:

• SwapController offers a robust solution for high-fidelity face swapping with superior identity and attribute control.
• The proposed methods effectively address the limitations of diffusion models in maintaining facial consistency.
• This framework advances the field of generative AI for realistic and controllable face manipulation.