Multimodal Fusion of Structural and Diffusion MRI for Intelligence Prediction

Multimodal neuroimaging fusion provides complementary insights into brain structure and/or function. However, effectively integrating features across modalities remains a challenging task. This study presents a deep learning-based multimodal fusion framework for predicting cognitive outcomes in children using data from the Adolescent Brain Cognitive Development (ABCD) Study. We focus on two imaging modalities: gray matter (GM) density derived from structural MRI and white matter fractional anisotropy (FA) derived from diffusion MRI. Modality-specific features were extracted using two separate convolutional neural networks (CNNs) and subsequently integrated through three fusion strategies: simple concatenation, multi-head attention, and transformer encoder-based fusion. We evaluated both single-modality and multimodal models to assess the added value of integration. Experimental results demonstrate that direct feature concatenation achieves the highest predictive performance, surpassing attention-based and transformer-based fusion approaches, with a test correlation of 0.44. Furthermore, we employed guided Grad-CAM to localize GM and FA regions contributing to intelligence prediction, providing interpretable neurological insights into the model's decision-making process.