Deep multimodal representations and classification of first-episode psychosis via live face processing

Area of Science:

• Neuroscience
• Psychiatry
• Machine Learning

Background:

• Schizophrenia is a severe psychiatric disorder with cognitive and social deficits.
• Early detection and treatment are crucial for reducing disease burden.
• Current diagnostic methods may not fully capture the complexity of early psychosis.

Purpose of the Study:

• To test if integrating multimodal data improves early psychosis prediction compared to unimodal recordings.
• To investigate the neural underpinnings of early psychosis using a novel framework.
• To develop a deep representation learning framework for classifying early psychosis.

Main Methods:

• Utilized multimodal data acquisition: functional near-infrared spectroscopy (fNIRS), electroencephalography (EEG), and facial features.
• Employed a novel deep representation learning framework, Neural-PRISM, for joint multimodal compressed representations.
• Analyzed neural correlates of social cognition during face-to-face interaction in first-episode psychosis (FEP) patients.

Main Results:

• Multimodal integration of fNIRS, EEG, and behavioral data significantly improved classification between controls and FEP individuals (10-20% enhancement).
• The Neural-PRISM framework effectively learned joint representations for describing, classifying, and predicting early psychosis severity.
• Geometric and topological features of brain activity trajectories revealed discriminatory neural characteristics in early psychosis.

Conclusions:

• Multimodal data integration offers a more robust approach for early psychosis detection and classification.
• The Neural-PRISM framework demonstrates potential for identifying neural signatures of early psychosis.
• Investigating social cognition through live interaction provides valuable insights into FEP neural correlates.