Multimodal depression detection based on an attention graph convolution and transformer.

Area of Science:

• Neuroscience
• Computer Science
• Artificial Intelligence

Background:

• Single-modal depression detection methods face limitations due to individual variability and noise.
• Existing multimodal approaches struggle with effective feature fusion from electroencephalogram (EEG) and speech signals.

Purpose of the Study:

• To develop and evaluate a multimodal depression detection model, MHA-GCN_ViT, integrating EEG and speech data.
• To enhance the accuracy and robustness of depression detection through advanced deep learning techniques.

Main Methods:

• Utilized discrete wavelet transform (DWT) for EEG feature extraction and graph convolutional networks (GCN) with multi-head attention for brain network analysis.
• Employed short-time Fourier transform (STFT) and vision transformers (ViT) to process spectral features from EEG and speech signals.
• Fused extracted features at the decision level for final depression classification.

Main Results:

• The MHA-GCN_ViT model achieved high performance metrics: 89.03% accuracy, 90.16% precision, 89.04% recall, and 88.83% F1 score on the MODMA dataset.
• Demonstrated significant improvement over traditional single-modal methods.
• Indicated robust performance and broad applicability for multimodal detection tasks.

Conclusions:

• The proposed MHA-GCN_ViT model offers a powerful and effective approach for multimodal depression detection.
• This method shows promise for diagnosing psychological and neurological disorders using combined EEG and speech data.