Related Experiment Video
Updated: Jul 9, 2026
11:28
Concurrent EEG and Functional MRI Recording and Integration Analysis for Dynamic Cortical Activity Imaging
Published on: June 30, 2018
11.6K
A multimodal vision transformer for interpretable fusion of functional and structural neuroimaging data
Yuda Bi1, Anees Abrol1, Zening Fu1
1Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia Tech, Emory, Atlanta, Georgia, USA.
Human Brain Mapping
|November 27, 2024
Summary
This study introduces MultiViT, a novel AI model that fuses structural and functional brain imaging data for improved schizophrenia diagnosis. The multimodal approach enhances diagnostic accuracy and identifies key brain regions associated with the disorder.
Area of Science:
- Neuroimaging
- Artificial Intelligence
- Brain Disorders
Background:
- Multimodal neuroimaging combined with deep learning AI offers advanced diagnostic capabilities for brain disorders.
- Integrating structural and functional imaging is crucial for comprehensive diagnosis, with structural data relevant for Alzheimer's and functional for schizophrenia.
Purpose of the Study:
- To develop and evaluate MultiViT, a novel deep learning model for fusing structural MRI and functional MRI data for schizophrenia diagnosis.
- To improve the accuracy of AI-based automated neuroimaging diagnostics for schizophrenia.
Main Methods:
- Developed MultiViT, a deep learning model using vision transformers and cross-attention to fuse 3D gray matter maps (structural MRI) with functional network connectivity matrices (functional MRI via ICA).
- Evaluated MultiViT's performance against unimodal and multimodal baselines.
Main Results:
- MultiViT achieved an Area Under the Curve (AUC) of 0.833, outperforming baseline models in schizophrenia classification.
- Identified critical brain regions associated with schizophrenia characteristics using MultiViT's attention maps and cross-attention mechanisms.
Conclusions:
- MultiViT demonstrates superior accuracy in AI-based automated neuroimaging diagnostics for schizophrenia.
- Pioneered an advanced data fusion approach by integrating functional network connectivity with structural gray matter data, enabling biomarker localization in 3D space.
