Predicting the Arousal and Valence Values of Emotional States Using Learned, Predesigned, and Deep Visual Features.

Area of Science:

• Computer Science
• Psychology
• Human-Computer Interaction

Background:

• Emotional state prediction is vital for personalized mental health treatments.
• The circumplex model categorizes emotions using arousal and valence dimensions.
• Virtual reality (VR) offers a platform for cognitive remediation exercises.

Purpose of the Study:

• To select and integrate machine learning models for predicting emotional states (arousal and valence) within a VR system.
• To enhance cognitive remediation therapies for individuals with mental health disorders by customizing treatments based on predicted emotional states.

Main Methods:

• Utilized the RECOLA database containing audio, video, and physiological recordings.
• Extracted deep visual features using the MobileNet-v2 convolutional neural network (CNN) trained on facial recordings.
• Employed optimizable ensemble regression and fused features/predictions for enhanced arousal and valence prediction.

Main Results:

• Achieved a root mean squared error (RMSE) of 0.1140 and Pearson's correlation coefficient (PCC) of 0.8000 for arousal prediction.
• Achieved an RMSE of 0.0790 and PCC of 0.7904 for valence prediction.
• Demonstrated successful prediction using half-face data as a proof of concept for VR integration.

Conclusions:

• Deep visual features extracted via CNNs are effective for predicting arousal and valence.
• The developed machine learning approach shows promise for real-time emotional state recognition in VR therapeutic applications.
• Accurate emotional state prediction can significantly improve the personalization and efficacy of VR-based mental health interventions.