Related Experiment Video
Updated: Dec 30, 2025
06:37
Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention
Published on: December 15, 2023
5.1K
Multimodal Emotion Recognition from Eye Image, Eye Movement and EEG Using Deep Neural Networks
Summary
Eye tracking glasses offer new features for emotion recognition, achieving 79.63% accuracy by combining eye images, eye movements, and electroencephalography (EEG) data using Bimodal Deep AutoEncoder (BDAE). This multimodal approach provides complementary information for classifying five emotions.
Area of Science:
- Multimodal emotion recognition
- Human-computer interaction
- Affective computing
Background:
- Electroencephalography (EEG) is complex for emotion recognition.
- New features are needed for accurate emotion classification.
- Eye tracking glasses offer potential for multimodal emotion recognition.
Purpose of the Study:
- Investigate the potential of eye tracking glasses for multimodal emotion recognition.
- Classify five emotions using eye images, eye movements, and EEG.
- Compare data fusion methods for emotion recognition.
Main Methods:
- Collected eye images, eye movements, and EEG data.
- Employed feature level fusion and Bimodal Deep AutoEncoder (BDAE) for data fusion.
- Compared four combinations of the three data modalities.
Main Results:
- The Bimodal Deep AutoEncoder (BDAE) achieved a 79.63% mean accuracy using three-modality fusion features.
- Fusion of eye image and eye movement data yielded a 71.99% classification accuracy.
- Analysis revealed complementary information across the three modalities.
Conclusions:
- Multimodal emotion recognition using eye tracking glasses, eye movements, and EEG is effective.
- BDAE fusion significantly enhances emotion classification accuracy.
- Eye tracking data provides valuable complementary information for emotion recognition.
