Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data

Area of Science:

• Psychology
• Computer Science
• Data Science

Background:

• Depression is a significant global mental health challenge.
• Traditional depression risk assessment methods lack objectivity and efficiency.
• Deep learning offers potential for improved, data-driven depression detection.

Purpose of the Study:

• To introduce a novel multimodal deep learning framework, the Audio, Video, and Text Fusion-Three Branch Network (AVTF-TBN), for depression risk detection.
• To develop and validate an emotion elicitation paradigm using distinct tasks (reading, interviewing) for gathering sensor-based depression data.
• To evaluate the efficacy of the AVTF-TBN model in detecting depression risk using fused auditory, visual, and textual data.

Main Methods:

• Developed the AVTF-TBN framework with separate branches for audio, video, and text data processing.
• Implemented an emotion elicitation paradigm with reading and interviewing tasks to collect multimodal sensor data.
• Utilized a multimodal fusion module to combine features from different modalities for predictive modeling.
• Evaluated model performance using metrics such as F1 Score, Precision, and Recall.

Main Results:

• The AVTF-TBN model achieved an F1 Score of 0.78, Precision of 0.76, and Recall of 0.81 when using data from both reading and interviewing tasks.
• Experimental results validated the emotion elicitation paradigm's effectiveness in generating relevant data.
• The study demonstrated the significant contribution of sensor-based data in depression risk detection.

Conclusions:

• The AVTF-TBN model shows high efficacy in detecting depression risk by integrating multimodal sensor data.
• The developed emotion elicitation paradigm is effective for collecting rich, sensor-based mental health data.
• This research highlights the potential of deep learning and multimodal data fusion for objective mental health assessment.