Detecting depression through speech and text from casual talks with fully automated virtual humans

Area of Science:

• Computational linguistics
• Psychiatry
• Human-computer interaction

Background:

• Depression is a growing global health concern with diagnostic challenges.
• Current methods often underestimate depression due to subjectivity.
• Novel approaches are needed for objective and early detection of depressive symptoms.

Purpose of the Study:

• To investigate voice-based markers for detecting depressive symptoms using virtual humans (VHs) in open-ended conversations.
• To develop and evaluate computational models for depression detection based on speech and text.
• To compare conversation-level and turn-level aggregation strategies for multimodal depression detection.

Main Methods:

• Collected speech and text data from 101 participants (42 with depressive symptoms) interacting with VHs (DEPTALK dataset).
• Utilized pre-trained transformer models for generating speech and text embeddings.
• Employed Extreme Gradient Boosting and Gated Recurrent Units for classification at conversation and turn levels, respectively.
• Evaluated unimodal (speech, text) and multimodal (fusion) models.

Main Results:

• Conversation-level aggregation with multimodal fusion achieved the highest F1 score (0.648).
• Speech-based analysis, particularly prosody, showed stronger depression cues than text alone.
• Turn-level aggregation improved text-based detection (F1=0.505) but multimodal performance did not exceed the best conversation-level model.
• Emotionally diverse interactions aggregated together enhanced depression detection.

Conclusions:

• Casual conversation analysis with VHs can reveal significant depression indicators, primarily through vocal prosody.
• Multimodal analysis combining speech and text improves depression detection accuracy.
• Virtual humans offer a promising avenue for objective and scalable screening of depressive symptoms.