Multi-modal emotion recognition using EEG and speech signals.

Area of Science:

• Human-Computer Interaction
• Affective Computing
• Biomedical Signal Processing

Background:

• Automatic Emotion Recognition (AER) is crucial for naturalistic Human-Machine Interactions (HMI).
• Emotions can be inferred from external cues (e.g., speech) and internal physiological signals (e.g., electroencephalogram - EEG).
• Existing multi-modal emotion databases are limited, necessitating new resources for advanced AER research.

Purpose of the Study:

• To construct and introduce the Multi-modal Emotion Database with four modalities (MED4).
• To evaluate the performance of AER algorithms using EEG and speech signals.
• To investigate the effectiveness of feature-level and decision-level fusion strategies for multi-modal emotion recognition.

Main Methods:

• Developed the MED4 database with synchronously recorded EEG, photoplethysmography, speech, and facial images from 32 participants.
• Utilized video stimuli to elicit happy, sad, angry, and neutral emotions in controlled (anechoic chamber) and naturalistic (noisy lab) environments.
• Implemented and compared baseline AER algorithms (I-vector + PLDA, TCN, ELM, MLP) and fusion strategies.

Main Results:

• Electroencephalogram (EEG) signals demonstrated higher accuracy in emotion recognition (up to 89.70%) compared to speech signals (up to 64.67%).
• Fusion strategies combining speech and EEG signals significantly improved AER accuracy, enhancing it by up to 25.92% over speech alone and 1.67% over EEG alone.
• Fusion methods enhanced the robustness of AER in noisy environments, outperforming single-modality approaches.

Conclusions:

• The MED4 database provides a valuable resource for advancing AER research.
• Multi-modal fusion, particularly combining EEG and speech, is highly effective for improving AER accuracy and robustness.
• The findings support the development of more sophisticated AER systems for naturalistic HMI.