A gender-emotion interaction multi-task network for depression recognition via transformer-based multimodal fusion

Area of Science:

• Computational linguistics
• Affective computing
• Machine learning for healthcare

Background:

• Depression is a prevalent mental health disorder with significant impact.
• Speech-based biomarkers offer a non-invasive method for depression detection.
• Existing methods often neglect the influence of gender and emotion on speech patterns.

Purpose of the Study:

• To develop an advanced depression recognition system.
• To incorporate gender and emotion interactions into speech-based depression detection.
• To improve the accuracy and robustness of depression recognition models.

Main Methods:

• Utilized a gender-emotion interaction multi-task network (G-EIMTNet).
• Employed transformer-based cross-modal fusion of Mel-spectrograms (CNN) and acoustic features (MRMR).
• Implemented a multi-task learning framework to model gender-emotion interactions.

Main Results:

• The proposed G-EIMTNet significantly outperformed baseline models on the AVEC2014 dataset.
• Achieved a 15.88% increase in accuracy and a 14.73% increase in F1 score.
• Ablation studies confirmed the effectiveness of multi-modal fusion and gender-emotion interaction.

Conclusions:

• The G-EIMTNet model demonstrates superior performance in speech-based depression recognition.
• Integrating gender and emotion context enhances the accuracy of depression detection systems.
• This approach offers a promising direction for developing more effective AI-driven mental health tools.