Multimodal emotion recognition using EEG and eye tracking data.

Area of Science:

• Neuroscience
• Computer Science
• Human-Computer Interaction

Background:

• Emotion recognition is crucial for understanding human-computer interaction.
• Integrating physiological signals offers a promising avenue for objective emotion detection.
• Existing methods often rely on single modalities, limiting recognition accuracy.

Purpose of the Study:

• To develop and evaluate a multimodal emotion recognition system.
• To investigate the effectiveness of combining electroencephalograph (EEG) signals and pupillary response.
• To compare feature-level and decision-level fusion strategies for improved emotion recognition.

Main Methods:

• Collected simultaneous EEG and eye-tracking data from five participants viewing 15 emotional film clips (positive, neutral, negative).
• Extracted emotion-relevant features from both EEG and eye-tracking data across 12 experimental conditions.
• Developed and applied feature-level and decision-level fusion models to integrate multimodal data.

Main Results:

• Individual modalities achieved average accuracies of 71.77% (EEG) and 58.90% (eye-tracking).
• Feature-level fusion improved accuracy to 73.59%.
• Decision-level fusion further enhanced accuracy to 72.98%, demonstrating the benefit of multimodal integration.

Conclusions:

• Combining EEG signals and eye-tracking data through fusion strategies significantly improves emotion recognition performance.
• Both feature-level and decision-level fusion are effective in leveraging multimodal physiological data for enhanced emotion detection.
• This multimodal approach offers a more robust and accurate method for objective emotion recognition.