Emotion Recognition Using Temporal Facial Skin Temperature and Eye-Opening Degree During Digital Content Viewing for...

Electroencephalography is a widely used method for emotion recognition. However, it requires specialized equipment, leading to high costs. Additionally, attaching devices to the body during such procedures may cause physical and psychological stress to participants. These issues are addressed in this study by focusing on physiological signals that are noninvasive and contact-free, and a generalized method for estimating emotions is developed. Specifically, the facial skin temperature and eye-opening degree of participants captured via infrared thermography and visible cameras are utilized, and emotional states are estimated while Japanese older adults view digital content. Emotional responses while viewing digital content are often subtle and dynamic. Additionally, various emotions occur during such situations, both positive and negative. Fluctuations in facial skin temperature and eye-opening degree reflect activities in the autonomic nervous system. In particular, expressing emotions through facial expressions is difficult for older adults; as such, emotional estimation using such ecological information is required. Our study results demonstrated that focusing on skin temperature changes and eye movements during emotional arousal and non-arousal using bidirectional long short-term memory yields an F1 score of 92.21%. The findings of this study can enhance emotion recognition in digital content, improving user experience and the evaluation of digital content.