E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition.

Area of Science:

• Computer Science
• Cognitive Science
• Affective Computing

Background:

• Emotion recognition is crucial in computer and cognitive science.
• Existing methods utilize diverse data sources like speech, facial expressions, EEG, and ET.
• The role of emotional stimuli is as important as the user's response.

Purpose of the Study:

• To develop a novel multimodal framework for emotion recognition.
• To integrate stimulus data with physiological signals for improved accuracy and robustness.
• To analyze users' psychological reactions and the stimuli eliciting them.

Main Methods:

• A multimodal approach synergizing stimulus data with physical and physiological signals.
• An emotional cognition experiment collecting EEG and ET data.
• Development of the Emotion-Multimodal Fusion Neural Network (E-MFNN) for data fusion.

Main Results:

• The E-MFNN framework effectively processes both stimulus and physiological data.
• Extensive comparisons demonstrated the framework's efficacy against existing models.
• Algorithmic variations within the framework were assessed, highlighting its robustness.

Conclusions:

• The proposed multimodal framework significantly enhances emotion recognition.
• Integrating stimulus and physiological data is key to accurate emotional cognition.
• The E-MFNN offers a robust and effective solution for multimodal emotion recognition.