MindLink-Eumpy: An Open-Source Python Toolbox for Multimodal Emotion Recognition.

Area of Science:

• Affective Computing
• Human-Computer Interaction
• Biomedical Signal Processing

Background:

• Emotion recognition is crucial for intelligent systems but suffers from low accuracy and subject dependency.
• Existing methods often rely on single modalities, limiting comprehensive emotional state assessment.

Purpose of the Study:

• To develop and validate an open-source software toolbox, MindLink-Eumpy, for enhanced emotion recognition.
• To integrate electroencephalogram (EEG) and facial expression data for multimodal emotion analysis.
• To address challenges of accuracy and subject dependence in emotion recognition research.

Main Methods:

• MindLink-Eumpy utilizes a multitask convolutional neural network (CNN) with transfer learning for facial expression analysis.
• EEG analysis employs both subject-dependent Support Vector Machine (SVM) and subject-independent Long Short-Term Memory (LSTM) models.
• Decision-level fusion combines predictions from CNN, SVM, and LSTM using weight enumerator and AdaBoost techniques.

Main Results:

• Multimodal approaches demonstrated superior performance over single-modal methods in both offline (DEAP, MAHNOB-HCI) and online experiments.
• Subject-dependent multimodal emotion recognition achieved 71.00% (valence) and 72.14% (arousal) accuracy.
• Subject-independent LSTM-based EEG analysis yielded 78.56% (valence) and 77.22% (arousal) accuracy.

Conclusions:

• The MindLink-Eumpy toolbox effectively enhances emotion recognition accuracy through multimodal data fusion.
• The developed system demonstrates feasibility and efficiency for real-world emotion recognition applications.
• Findings highlight the potential of integrating physiological and visual cues for robust affective computing.