Wireless Sensing Technology Combined with Facial Expression to Realize Multimodal Emotion Recognition

Area of Science:

• Neuroscience
• Psychology
• Medicine
• Artificial Intelligence
• Signal Processing

Background:

• Emotions profoundly influence physical and mental health, making emotion recognition a critical research area.
• Existing methods for emotion recognition often rely on visual cues or physiological signals independently.
• Millimeter-wave (MMW) radar offers a non-invasive method to capture subtle physiological signals like heartbeat and respiration.

Purpose of the Study:

• To develop and evaluate a novel deep learning model for accurate emotion recognition.
• To integrate physiological data from MMW radar with facial expression analysis.
• To compare the performance of the proposed model against traditional machine learning algorithms and single deep learning approaches.

Main Methods:

• Raw signals from millimeter-wave radar were preprocessed to extract high-quality heartbeat and respiration data.
• A hybrid deep learning architecture was designed, combining a convolutional neural network (CNN) for spatial feature extraction and a gated recurrent unit (GRU) neural network for temporal sequence modeling.
• The model was trained and validated using a dataset that includes both MMW radar signals and human face expression images.

Main Results:

• The proposed deep learning model achieved a person-dependent recognition accuracy of 84.5%.
• In person-independent experiments, the model demonstrated a recognition accuracy of 74.25%.
• Experimental results indicate that the integrated deep learning model significantly outperforms traditional machine learning algorithms and single deep learning models.

Conclusions:

• The fusion of MMW radar-derived physiological signals and facial expressions via a CNN-GRU deep learning model offers a promising approach for robust emotion recognition.
• This method provides a more comprehensive understanding of emotional states by leveraging multi-modal data.
• The findings suggest potential applications in healthcare, human-computer interaction, and mental well-being monitoring.