An interpretable cross-attentive multi-modal MRI fusion framework for schizophrenia identification.

Area of Science:

• Neuroimaging
• Artificial Intelligence in Medicine
• Psychiatric Disorders

Background:

• Functional MRI (fMRI) and structural MRI (sMRI) provide complementary brain data.
• Integrating fMRI and sMRI for schizophrenia identification is challenging due to data heterogeneity and difficulties in modeling inter-modal interactions.
• Existing methods often fail to effectively capture the complex relationships between different imaging modalities.

Purpose of the Study:

• To develop an accurate and interpretable framework for multimodal brain imaging analysis in schizophrenia.
• To effectively model the interaction between functional and structural brain imaging data for improved schizophrenia classification.
• To enhance the interpretability of machine learning models in neuroimaging by highlighting salient features.

Main Methods:

• Proposed a Cross-Attentive Multi-modal Fusion (CAMF) framework utilizing self-attention for intra-modal pattern recognition and cross-attention for inter-modal relationship learning.
• Introduced a gradient-guided score-class activation map for enhanced model interpretability.
• Evaluated the framework on multi-modal brain imaging datasets from four schizophrenia study cohorts.

Main Results:

• CAMF significantly improved the accuracy of schizophrenia classification compared to existing methods.
• The framework successfully identified functional networks and anatomical regions consistent with known schizophrenia biomarkers.
• Demonstrated enhanced interpretability by visualizing salient features contributing to classification decisions.

Conclusions:

• CAMF offers an accurate and interpretable approach for integrating fMRI and sMRI data in schizophrenia research.
• The framework provides novel insights into schizophrenia-related alterations by effectively fusing multimodal neuroimaging information.
• CAMF represents a significant advancement in applying deep learning for multimodal brain analysis in psychiatric disorders.