Multi-branch convolutional neural network with cross-attention mechanism for emotion recognition

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Neuroscience

Background:

• Emotion recognition has broad applications in education, marketing, and healthcare.
• Accurate emotion recognition from multimodal data remains a challenge.
• Existing models often struggle with effective feature extraction and fusion.

Purpose of the Study:

• To propose a novel Multi-Branch Convolutional Neural Network with Cross-Attention (MCNN-CA) for accurate emotion recognition.
• To automate feature extraction from multimodal data and fuse diverse feature maps.
• To improve the inter-correlation of features within and across different data modalities.

Main Methods:

• Developed a MCNN-CA model utilizing various Convolutional Neural Networks for feature extraction.
• Implemented a feature fusion module with a channel-efficient attention mechanism.
• Evaluated the model on EEG emotion recognition datasets (SEED, SEED-IV) and multimodal EEG-text data (ZuCo).

Main Results:

• The MCNN-CA model demonstrated superior performance in emotion recognition tasks.
• Achieved high accuracy, precision, recall, and F1-score compared to contemporary methods.
• Effective fusion of single-mode and cross-mode features was confirmed.

Conclusions:

• The proposed MCNN-CA model offers a robust and accurate solution for emotion recognition.
• The cross-attention mechanism and feature fusion module are key to its enhanced performance.
• This approach holds significant potential for advancing affective computing and human-computer interaction.