Do Deepfakes Adequately Display Emotions? A Study on Deepfake Facial Emotion Expression

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Advancements in AI enable creation of hyper-realistic deepfakes, often using face-swapping without consent.
• Emotional expression is crucial for human communication, making its study in deepfakes relevant.

Purpose of the Study:

• To conduct an in-depth study of facial emotional expression transfer in deepfakes.
• To analyze the accuracy of emotional expression replication in face-swap deepfakes.

Main Methods:

• Utilized a well-known face swap-based deepfake database.
• Extracted individual frames (photograms) from original and deepfake videos.
• Analyzed facial emotional expressions in both original and generated deepfake recordings.

Main Results:

• Emotional expressions are not adequately transferred from original recordings to deepfakes.
• Significant variability in emotion and performer expressiveness was detected between original and fake videos.

Conclusions:

• Current deepfake generation methods show limitations in replicating authentic emotional expressions.
• Performer emotion expressiveness is a critical factor to consider for improving deepfake generation and detection techniques.