Self-Supervised Guided Modality Disentangled Representation Learning for Multimodal Sentiment Analysis and...

Area of Science:

• Artificial Intelligence
• Computational Psychiatry
• Machine Learning

Background:

• Chronic mental disorders pose a growing challenge, necessitating advanced diagnostic and treatment tools.
• Multimodal sentiment analysis (MSA) offers a promising avenue for improving mental health assessments by integrating diverse data types.

Purpose of the Study:

• To develop an advanced MSA model that addresses modality heterogeneity and enhances diagnostic accuracy for mental disorders.
• To leverage disentangled representation learning and self-supervised learning for robust feature extraction and fusion.

Main Methods:

• Employed disentangled representation learning guided by self-supervised learning to generate pseudo unimodal labels and prevent meaningless feature acquisition.
• Introduced a text-centric fusion mechanism to effectively mitigate noise and redundant information, creating a comprehensive multimodal representation.
• Evaluated the model on three public MSA benchmark datasets and a private dataset for schizophrenia counseling.

Main Results:

• Achieved state-of-the-art performance across multiple metrics on benchmark datasets, outperforming existing related works.
• Demonstrated significant progress in schizophrenia assessment on a real-world dataset, surpassing previous methodologies.
• The proposed self-supervised learning approach effectively guided modality-specific representation learning.

Conclusions:

• The developed MSA model effectively handles modality heterogeneity and improves sentiment analysis for mental health applications.
• The approach shows significant potential for real-world clinical applications, particularly in the assessment of schizophrenia.
• This work advances the field of multimodal sentiment analysis for mental disorder diagnosis and treatment.