MSER: an emotion recognition method based on multi-signal information fusion

Emotion recognition usually refers to the identification of people's emotional states through facial expressions, behaviors, etc. Introducing the emotion recognition method that fuses physiological signals has become more and more necessary to cope with the limitations from traditional emotion recognitions, which are highly subjective and often ignore the physiological information induced by emotional expression.

In this work, Multi-Signals Emotion Recognition (MSER) and its variant based Attention Mechanism (MSER-Att) are proposed to improve the effectiveness and accuracy of physiology-based emotion recognition. The basic structure of MSER and MSER-Att include VGG network and Ranger optimizer, while MSER-Att integrates with the attention mechanism. The two proposed structure fill the gap in the application of the VGG network and Ranger optimizer in this field. First, four different physiological signals from WESAD dataset are combined to form a new dataset. Then MSER and MSER-Att are used to extract features and recognize emotions. The training process is optimized by Ranger with early stopping strategy and tenfold cross validation.

Comparing the results with other methods of emotion recognition, MSER-Att yields an accuracy of 95.41% and an F1 score of 98.54%, with which each emotion classification obtains PPV and TPR for more than 96%, even up to 100%.

The findings in this article show that integrating various physiological signals with the help of attention mechanisms and the VGG network can enhance the possibilities of multi-signal emotion recognition systems.