Self-Learning Multimodal Emotion Recognition Based on Multi-Scale Dilated Attention

Area of Science:

• Affective computing
• Multimodal machine learning
• Biomedical signal processing

Background:

• Emotion recognition is challenging due to the complex interplay of psychological and physiological factors.
• Single-modality models often lack robustness in emotion detection.
• Integrating diverse data sources like EEG and facial expressions can enhance recognition accuracy.

Purpose of the Study:

• To develop a robust multimodal emotion recognition framework.
• To improve performance by integrating electroencephalogram (EEG) signals and facial expressions.
• To enhance the adaptive fusion of multimodal data for accurate emotion analysis.

Main Methods:

• A Multi-Scale Dilated Attention Convolution (MSDAC) network was used for facial expression analysis.
• EEG signals were processed into multidimensional time-frequency-spatial representations.
• A self-learning decision-level fusion strategy adaptively combined information from both modalities.

Main Results:

• The facial analysis branch achieved high accuracies across multiple datasets (e.g., 99.69% on CK+).
• The multimodal model demonstrated strong performance on the DEAP dataset, reaching 98.66% for valence and 97.49% for arousal.
• The proposed framework significantly improved emotion recognition capabilities.

Conclusions:

• The multimodal framework effectively integrates EEG and facial expression data.
• Improved facial feature extraction and adaptive fusion enhance emotion recognition robustness.
• Combining EEG and facial information offers a promising approach for comprehensive emotion analysis.