Learning Spatial-Temporal Coherent Correlations for Speech-Preserving Facial Expression Manipulation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Speech-preserving facial expression manipulation (SPFEM) traditionally requires paired training data, limiting real-world applications.
• Existing methods struggle to modify facial emotions while preserving accurate lip-sync for spoken content.

Purpose of the Study:

• To develop a novel algorithm for SPFEM that does not rely on paired training data.
• To enable realistic manipulation of facial expressions while maintaining accurate mouth animations for speech.

Main Methods:

• Proposed a spatial-temporal coherent correlation learning (STCCL) algorithm.
• Developed spatial and temporal coherent correlation metrics to supervise expression manipulation and preserve facial animation.
• Incorporated a correlation-aware adaptive strategy to focus on challenging facial regions.

Main Results:

• The STCCL algorithm effectively manipulates facial expressions while preserving speech-related mouth movements.
• Experimental results across various datasets validate the proposed method's effectiveness.
• Demonstrated the utility of spatial-temporal correlations for unsupervised SPFEM.

Conclusions:

• The STCCL algorithm offers a promising solution for real-world SPFEM applications by overcoming the limitations of paired training data.
• The findings highlight the importance of modeling spatial-temporal correlations for accurate facial animation preservation.
• This work advances the field of facial expression manipulation with improved speech preservation.