Predicting depression by using a novel deep learning model and video-audio-text multimodal data.

Area of Science:

• Computational psychiatry
• Machine learning in healthcare
• Multimodal data analysis

Background:

• Depression is a widespread mental health issue with subjective diagnostic methods.
• Current diagnostic tools like questionnaires and interviews have limitations in objectivity and consistency.
• There is a need for more accurate and reliable methods for depression evaluation.

Purpose of the Study:

• To introduce the Integrative Multimodal Depression Detection Network (IMDD-Net), a novel deep-learning framework.
• To enhance the accuracy of depression evaluation by integrating local and global features from multimodal data.
• To provide a holistic analysis of depressive symptoms using video, audio, and text cues.

Main Methods:

• The IMDD-Net framework uses Kronecker product for multimodal fusion, enabling deep interactions.
• Audio features include Mel Frequency Cepstrum Coefficient (MFCC) and extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS).
• Video data is processed by TimeSformer for temporal features, and text data uses a pre-trained BERT model.

Main Results:

• The IMDD-Net achieved state-of-the-art performance on the AVEC 2014 dataset.
• It demonstrated a Root Mean Square Error (RMSE) of 7.55 and Mean Absolute Error (MAE) of 5.75 in predicting Beck Depression Inventory-II (BDI-II) scores.
• The model achieved an accuracy of 0.79 in classifying potential depression subjects.

Conclusions:

• The IMDD-Net shows robustness and precision in depression prediction.
• Integrating local and global features across multiple modalities is crucial for accurate depression assessment.
• This multimodal deep-learning approach offers a promising advancement in mental health diagnostics.