Machine Learning-Based Ear Thermal Imaging for Emotion Sensing

Area of Science:

• Psychophysiology
• Machine Learning
• Biomedical Engineering

Background:

• Thermal imaging offers non-contact, light-independent measurement of skin temperature, reflecting autonomic nervous system activity.
• Previous studies suggest a linear link between ear temperature and emotional arousal, but nonlinear relationships remain unexplored.
• Accurate emotion sensing is crucial for various applications, including mental health monitoring and human-computer interaction.

Purpose of the Study:

• To investigate nonlinear relationships between ear thermal changes and subjective emotional arousal.
• To compare the performance of machine learning models against linear regression for emotion sensing using thermal imaging.
• To identify specific ear regions associated with emotional arousal through model interpretation.

Main Methods:

• Reanalyzed thermal ear images and self-reported arousal ratings from participants watching emotion-eliciting films.
• Employed linear regression, random forest, and ResNet-50 convolutional neural network models.
• Evaluated models using mean squared error and correlation coefficients; interpreted the best model with Gradient-weighted Class Activation Mapping and Shapley additive explanation.

Main Results:

• Both machine learning models significantly outperformed linear regression in predicting arousal ratings.
• The ResNet-50 deep learning model demonstrated superior performance compared to the random forest model.
• Model interpretation revealed nonlinear associations between temperature variations in specific ear regions and subjective arousal levels.

Conclusions:

• Ear thermal imaging, when combined with advanced machine learning techniques like deep learning, shows significant promise for accurate emotion sensing.
• Nonlinear modeling approaches are essential for capturing the complex relationship between physiological signals and subjective emotional states.
• This study highlights the potential of thermal imaging as a non-invasive tool for understanding and quantifying human emotions.