Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning.

Area of Science:

• Physiological computing
• Affective computing
• Machine learning for emotion recognition

Background:

• Traditional emotion recognition relies on facial expressions, which may not reflect true emotional states.
• Subtle or hidden emotions, especially during passive viewing, are challenging to detect with expression-based methods.
• Remote sensing of physiological signals offers a non-intrusive alternative for emotion classification.

Purpose of the Study:

• To improve a remote emotion classification approach using physiological signals from facial videos.
• To enhance heart rate estimation and heartbeat detection for more accurate emotion analysis.
• To achieve better emotion classification accuracy using deep learning with only RGB camera data.

Main Methods:

• Utilized short facial video data from 110 participants passively viewing emotion-eliciting videos.
• Employed remote sensing of transdermal cardiovascular spatiotemporal facial patterns.
• Applied machine learning, specifically an EfficientNet-B0 model, to classify five emotion types (amusement, disgust, fear, sexual arousal, no emotion).
• Incorporated improvements in skin segmentation for heart rate estimation and heartbeat peak/trough detection.

Main Results:

• An EfficientNet-B0 model achieved an overall average accuracy of 47.36% for emotion classification.
• This accuracy was obtained using only a single spatiotemporal feature map from an RGB camera.
• The study demonstrated the feasibility of remote emotion sensing via physiological patterns.

Conclusions:

• Remote sensing of physiological patterns from facial videos is a viable approach for emotion recognition.
• Deep learning models, like EfficientNet-B0, can effectively classify emotions using physiological data from RGB cameras.
• Further research can refine these methods for more accurate and robust emotion detection.