A Sentiment Pre-trained Text-Guided Multimodal Cross-Attention Transformer for Improved Depression Detection

Area of Science:

• Computational psychiatry
• Machine learning for mental health
• Multimodal affective computing

Background:

• Depression is a prevalent mental health disorder necessitating advanced automated detection.
• Single-modality approaches struggle with the complexity of depression, driving interest in multimodal analysis.
• Existing transformer methods for depression detection are often text-dominant, limiting their scope.

Purpose of the Study:

• To propose a novel Text-Guided Multimodal Cross-Attention Transformer (TGMCT) for enhanced depression detection.
• To improve cross-modal interactions among text, audio, and video data.
• To leverage pre-trained encoders for better emotion-related feature extraction.

Main Methods:

• Developed a transformer architecture incorporating cross-attention mechanisms for multimodal fusion.
• Pre-trained encoders on a large sentiment dataset to capture nuanced emotional expressions.
• Evaluated the TGMCT model on the AVEC2019 benchmark dataset.

Main Results:

• The proposed TGMCT model achieved superior performance in depression detection compared to existing methods.
• Enhanced cross-modal fusion significantly improved the accuracy of depression identification.
• Pre-training on sentiment data proved effective in identifying subtle sentiment shifts indicative of depression.

Conclusions:

• The TGMCT model offers a more effective approach to automated depression detection by leveraging multimodal data.
• Cross-modal attention and sentiment-aware pre-training are key to improving depression detection accuracy.
• This research advances the field of computational psychiatry with a state-of-the-art multimodal fusion technique.