E-TIME: Emotion Trend Inspired Multi-task Sparse Mask Neural Network for Multimodal Emotion Recognition

Emotion recognition based on physiological signals is important for both the diagnosis of mental disorders and human-computer interaction. Although numerous existing methods have achieved promising performance in recognizing emotions, several challenges still remain open: 1) As an important attribute, the trend of emotion isn't being captured explicitly. 2) Fixed feature extraction scales cannot accommodate the fact that the periods of multimodal physiological signals vary along with emotions. To address these challenges, we propose a multi-task sparse mask neural network called E-TIME for recognizing emotions and trends of emotions simultaneously. Specifically, E-TIME consists of multimodal physiological signal representation generators and the multi-task trend learning component. To cope with varying periods of physiological signals, the multimodal representation generator adaptively discovers the optimal feature extraction scale based on dynamic sparse mask convolution. The multi-task trend learning component achieves trend capture and accurate emotion recognition by constructing emotion trend recognition tasks at different moments and utilizing representations of the trend task in the emotion recognition task. Extensive experiments on two real-world datasets demonstrate that our proposed method achieves better performance than the superior baseline.