Multi-Modal Emotion Recognition Using EEG and Eye Tracking Features

Area of Science:

• Affective computing and human-computer interaction.
• Neuroscience and signal processing.

Background:

• Multi-modal emotion recognition leverages physiological signals like EEG, ECG, GSR, and eye-tracking.
• The SEED V dataset is a benchmark for evaluating emotion recognition models.

Purpose of the Study:

• To develop a Convolutional Neural Network (CNN) based multi-modal emotion recognition model using EEG and eye-tracking data.
• To investigate the impact of different time window sizes for EEG feature extraction on emotion recognition performance.
• To achieve state-of-the-art results on the SEED V dataset.

Main Methods:

• EEG signals were converted into a 2D image format to retain spatial information.
• Differential Entropy (DE) was used for EEG feature extraction across varying time windows (1s and 4s).
• A simple CNN architecture was employed for multi-modal fusion of EEG and eye-tracking features.

Main Results:

• The proposed model achieved a mean accuracy of 0.935 ± 0.038 using a 1-second EEG processing window in Leave One Subject Out Validation.
• This 1-second window significantly outperformed the 4-second window, demonstrating the advantage of shorter processing durations.
• The model achieved state-of-the-art performance on the SEED V dataset.

Conclusions:

• Shorter time windows (1 second) are crucial for effective EEG feature processing in emotion recognition tasks.
• The developed multi-modal CNN model shows high efficacy for emotion recognition using EEG and eye-tracking.
• This research highlights the importance of temporal feature resolution in affective computing.