TSFFM: Depression detection based on latent association of facial and body expressions

Depression is a prevalent mental disorder worldwide. Early screening and treatment are crucial in preventing the progression of the illness. Existing emotion-based depression recognition methods primarily rely on facial expressions, while body expressions as a means of emotional expression have been overlooked. To aid in the identification of depression, we recruited 156 participants for an emotional stimulation experiment, gathering data on facial and body expressions. Our analysis revealed notable distinctions in facial and body expressions between the case group and the control group and a synergistic relationship between these variables. Hence, we propose a two-stream feature fusion model (TSFFM) that integrates facial and body features. The central component of TSFFM is the Fusion and Extraction (FE) module. In contrast to conventional methods such as feature concatenation and decision fusion, our approach, FE, places a greater emphasis on in-depth analysis during the feature extraction and fusion processes. Firstly, within FE, we carry out local enhancement of facial and body features, employing an embedded attention mechanism, eliminating the need for original image segmentation and the use of multiple feature extractors. Secondly, FE conducts the extraction of temporal features to better capture the dynamic aspects of expression patterns. Finally, we retain and fuse informative data from different temporal and spatial features to support the ultimate decision. TSFFM achieves an Accuracy and F1-score of 0.896 and 0.896 on the depression emotional stimulus dataset, respectively. On the AVEC2014 dataset, TSFFM achieves MAE and RMSE values of 5.749 and 7.909, respectively. Furthermore, TSFFM has undergone testing on additional public datasets to showcase the effectiveness of the FE module.