Multimodal Depression Detection Through Conversational Interactions with an Emotion-Aware Social Robot: Pilot Study

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Mental Health Technology

Background:

• Depression impacts over 300 million globally, posing a significant disease burden.
• Traditional depression screening methods are often impractical for widespread use.
• Existing AI approaches for depression detection face challenges with data scarcity, adaptability, and computational cost.

Purpose of the Study:

• To introduce DEPRESAR-Fusion, a lightweight multimodal framework for depression detection in social assistive robots (SARs).
• To improve depression detection accuracy in natural conversations, addressing data scarcity and computational efficiency.
• To enable emotion-aware SARs for enhanced mental health monitoring.

Main Methods:

• DEPRESAR-Fusion integrates acoustic, linguistic, and visual features with large language models for adaptive conversation.
• Emotion induction via evocative videos was used to enhance participant emotional expression.
• Data augmentation techniques, including public corpora and synthetic data, addressed data scarcity.
• The framework was evaluated on clinical datasets for binary classification and PHQ-8 regression.

Main Results:

• Emotional stimuli significantly increased participant expressiveness and improved model performance.
• DEPRESAR-Fusion achieved state-of-the-art results in both depression classification and PHQ-8 regression.
• The system demonstrated superior performance compared to previous multimodal baselines.
• The lightweight architecture is suitable for real-time deployment on SARs.

Conclusions:

• DEPRESAR-Fusion enables accurate and scalable depression detection in naturalistic SAR interactions.
• The approach combines emotion induction, data augmentation, and multimodal fusion effectively.
• This highlights the potential of SARs as nonintrusive tools for proactive mental health support.