Multimodal depression recognition and analysis: Facial expression and body posture changes via emotional stimuli

Clinical studies have shown that facial expressions and body posture in depressed patients differ significantly from those of healthy individuals. Combining relevant behavioral features with artificial intelligence technology can effectively improve the efficiency of depression detection, thereby assisting doctors in early identification of patients. This study aims to develop an end-to-end multimodal recognition model combining facial expressions and body posture via deep learning techniques, enabling rapid preliminary screening of depression.

We invited 146 subjects (73 in the patient group and 73 in the control group) to participate in an emotion-stimulus experiment for depression recognition. We focused on differentiating depression patients from the control group by analyzing changes in body posture and facial expressions under emotional stimuli. We first extracted images of body position and facial emotions from the video, then used a pre-trained ResNet-50 network to extract features. Additionally, we analyzed facial expression features using OpenFace for sequence analysis. Subsequently, various deep learning frameworks were combined to assess the severity of depression.

We found that under different stimuli, facial expression units AU04, AU07, AU10, AU12, AU17, and AU26 had significant effects in the emotion-stimulus experiment, with these features generally being negative. The decision-level fusion model based on facial expressions and body posture achieved excellent results, with the highest accuracy of 0.904 and an F1 score of 0.901.

The experimental results suggest that depression patients exhibit predominantly negative facial expressions. This study validates the emotion-stimulus experiment, demonstrating that combining facial expressions and body posture enables accurate preliminary depression screening.