Development of Machine-Learning-Based Facial Thermal Image Analysis for Dynamic Emotion Sensing

Area of Science:

• Thermography
• Affective Computing
• Machine Learning

Background:

• Facial thermal patterns correlate with emotional states.
• Previous linear analysis of facial thermal data (regions of interest) may miss complex, nonlinear information.
• Accurate emotion sensing is valuable for various applications.

Purpose of the Study:

• To investigate machine learning (ML) for pixel-level analysis of facial thermal images.
• To estimate dynamic emotional arousal ratings using ML.
• To compare ML performance against traditional linear regression models.

Main Methods:

• Collected facial thermal data from 20 participants viewing emotion-eliciting films.
• Utilized ML models: random forest regression, support vector regression, ResNet-18, and ResNet-34.
• Interpreted nonlinear relationships using saliency maps and integrated gradients for ResNet-34.

Main Results:

• ML models significantly outperformed linear regression in estimating arousal.
• ResNet-18 and ResNet-34 showed superior performance.
• Nonlinear associations were found between arousal and temperature changes in the nose tip, forehead, and cheeks.

Conclusions:

• ML-based pixel-level analysis of facial thermal images is effective for estimating emotional arousal.
• Nonlinear thermal patterns provide valuable insights into emotional states.
• Potential applications include non-invasive emotion sensing in mental health, education, and human-computer interaction.