A multi-modal approach for identifying schizophrenia using cross-modal attention.

Area of Science:

• Neuroscience
• Computer Science
• Psychiatry

Background:

• Schizophrenia classification often relies on limited data modalities.
• Distinguishing schizophrenia, particularly positive symptoms, requires nuanced analysis of communication patterns.

Purpose of the Study:

• To develop and evaluate a multi-modal system for classifying schizophrenia using audio, video, and text data.
• To enhance the accuracy of schizophrenia detection by integrating diverse communication features.

Main Methods:

• Extracted low-level facial action units (video) and vocal tract variables (audio).
• Computed high-level coordination features from audio-video data.
• Utilized context-independent text embeddings from speech transcriptions.
• Developed a fusion model combining segment-to-session classifiers (audio/video) with a Hierarchical Attention Network (HAN) for text, incorporating cross-modal attention.

Main Results:

• The proposed multi-modal system demonstrated superior performance compared to prior state-of-the-art methods.
• Achieved an 8.53% improvement in weighted average F1 score.

Conclusions:

• Multi-modal analysis of human communication offers a powerful approach for schizophrenia classification.
• The developed system shows significant potential for improving diagnostic accuracy in schizophrenia research.