Attention-based 3D convolutional recurrent neural network model for multimodal emotion recognition.

Area of Science:

• Artificial Intelligence
• Biomedical Engineering
• Cognitive Science

Background:

• Multimodal emotion recognition is crucial for human-computer interaction and intelligent healthcare.
• Integrating diverse human modalities (e.g., visual, EEG) for emotion computation remains challenging.
• Existing methods often struggle to effectively fuse information from different modalities.

Purpose of the Study:

• To propose a novel 3D convolutional recurrent neural network (3FACRNN) model for enhanced multimodal emotion recognition.
• To investigate the fusion of visual and electroencephalogram (EEG) data using an attention mechanism.
• To improve the accuracy and stability of emotion recognition systems.

Main Methods:

• Developed a 3FACRNN network comprising a visual network (CNN-TCN) and an EEG network.
• Integrated band, spatial, and temporal information in the EEG network with 3D feature building.
• Employed band attention and self-attention modules within a convolutional recurrent neural network (CRNN).
• Utilized a multi-task loss function (Lc) to align intermediate feature vectors from visual and EEG modalities.

Main Results:

• The 3FACRNN model achieved high recognition accuracies on DEAP and MAHNOB-HCI datasets (e.g., 96.75% for arousal, 96.86% for valence on DEAP).
• Results indicate that high-frequency gamma bands (31-50 Hz) are particularly relevant for emotion recognition.
• The proposed method outperformed state-of-the-art multimodal emotion recognition approaches.

Conclusions:

• Multimodal fusion of facial video and EEG signals enhances emotion recognition network stability and accuracy.
• The attention mechanism effectively leverages relevant information from different frequency bands and modalities.
• Future work will explore sparse matrix methods and deep convolutional networks for further performance improvements.