A gender-emotion interaction multi-task network for depression recognition via transformer-based multimodal fusion

Depression is characterized by high prevalence, high recurrence, high disability and high mortality, which seriously affects people's work and life. Among various behavioral biomarkers, speech-based features have gained increasing attention in depression detection due to their non-invasive nature, affordability, and rich capacity for conveying affective states. However, conventional depression recognition approaches rely solely on unimodal acoustic representations and largely overlook the influence of emotion and gender. To address this limitation, this study proposed a gender-emotion interaction multi-task network(G-EIMTNet) for depression recognition via transformer-based cross modal fusion. In the feature fusion stage, the deep representations of Mel-spectrograms were extracted using convolutional neural networks(CNN), and then the Maximum Correlation Minimum Redundancy (MRMR) algorithm was employed to select acoustic higher-order statistical features that were highly correlated with emotions and depressive states. These two types of features were then fused through the transformer attention mechanism. In the depression recognition stage, a depression recognition network for the interaction between gender and emotion was constructed based on a multi-task framework. Experiments on the AVEC2014 dataset showed that this approach outperformed the baseline model by 15.88% and 14.73% in accuracy and F1 score, respectively. Ablation experiments verify the effectiveness of multi-modal fusion and gender-emotion interaction.