Enhancing emotion recognition in virtual reality: a multimodal dataset and a temporal emotion detector

Area of Science:

• Psychophysiology
• Human-Computer Interaction
• Affective Computing

Background:

• Emotion recognition is crucial for various applications, with virtual reality (VR) offering immersive environments.
• Existing VR multimodal emotion datasets are limited, hindering progress in accurate emotion recognition.
• Current multimodal approaches struggle with noise, individual variability, and generalization.

Purpose of the Study:

• To address the scarcity of VR multimodal emotion datasets and improve emotion recognition accuracy.
• To construct a new VR dataset and develop a robust multimodal emotion recognition model.
• To evaluate the performance of the proposed model on existing and newly collected datasets.

Main Methods:

• Developed a VR experimental environment with 10 emotion-eliciting video clips based on the PAD model.
• Collected electrodermal activity, eye-tracking, and questionnaire data from 38 participants.
• Proposed the Multi-Modal Temporal Emotion Detector (MMTED) model integrating baseline calibration and multimodal fusion.

Main Results:

• The MMTED model achieved high recognition accuracies: 85.52% on the VREED dataset, 89.27% on the new dataset, and 85.29% on the combined dataset.
• The newly collected dataset expands available VR-based multimodal emotion resources.
• Demonstrated the effectiveness of multimodal fusion of electrodermal and eye-tracking signals for emotion recognition.

Conclusions:

• The MMTED model offers a robust solution for emotion recognition in VR environments.
• The new dataset and model contribute significantly to advancing VR-based affective computing.
• Multimodal physiological signal processing and emotion modeling in VR remain a promising research area.