Visual Scene-Aware Hybrid and Multi-Modal Feature Aggregation for Facial Expression Recognition.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep learning has advanced facial expression recognition (FER), but current methods struggle with real-world video variations like illumination and head poses.
• Existing FER techniques often fail in uncontrolled environments due to limitations in training data and model robustness.

Purpose of the Study:

• To develop a robust FER system capable of handling variations in wild environments.
• To improve FER performance by proposing a novel multi-modal neural network architecture.
• To effectively fuse image and facial landmark information for enhanced recognition accuracy.

Main Methods:

• A frame substitution module was developed to optimize recurrent neural network (RNN) performance by replacing less informative frame features.
• Facial landmark features were extracted using an inter-frame correlation method.
• A novel multi-modal fusion technique was proposed, utilizing attention mechanisms to integrate video and landmark data at the feature level.

Main Results:

• The proposed method achieved 51.4% accuracy on the challenging AFEW dataset.
• The system demonstrated high accuracy on controlled datasets, reaching 98.5% on CK+ and 81.9% on MMI.
• The novel approach significantly outperformed existing state-of-the-art FER networks.

Conclusions:

• The proposed multi-modal fusion approach significantly enhances facial expression recognition accuracy, particularly in uncontrolled environments.
• The frame substitution and landmark feature extraction methods contribute to improved robustness against varying conditions.
• This research offers a promising direction for developing more reliable and accurate FER systems for real-world applications.