Multimodal Depression Recognition via Mutual Information Maximization Joint With Multi-Task Learning

Area of Science:

• Psychiatry
• Computer Science
• Artificial Intelligence

Background:

• Depression is a significant mental health disorder impacting individuals and society.
• Multimodal data (vision, audio, text) is crucial for accurate depression diagnosis.
• Existing methods often overlook feature enhancement and fusion within and across modalities.

Purpose of the Study:

• To establish a Chinese Multimodal Depression Corpus (CMD-Corpus) for research.
• To propose a novel multimodal depression recognition framework (MIMML).
• To enhance feature representation and fusion for improved depression detection.

Main Methods:

• Developed the Chinese Multimodal Depression Corpus (CMD-Corpus) with clinical expert assistance.
• Proposed the Mutual Information Maximization with Multi-task Learning (MIMML) framework.
• Implemented modality-invariance enhancement via mutual information maximization.
• Utilized multi-task learning for improved single-modality representation.
• Employed a gated structure with bidirectional GRUs and CNNs for multimodal feature fusion.

Main Results:

• The MIMML framework effectively enhances feature representation and fusion.
• Achieved 84% accuracy on the DAIC-WOZ dataset.
• Achieved 89% accuracy on the self-collected CMD-Corpus dataset.
• Demonstrated significant improvements in depression recognition accuracy.

Conclusions:

• The proposed MIMML framework shows high effectiveness in multimodal depression recognition.
• The CMD-Corpus provides a valuable resource for future depression research.
• Enhanced feature representation and fusion are key to improving diagnostic accuracy for mental health disorders.