Multimodal Fusion of Structural and Diffusion MRI for Intelligence Prediction

Area of Science:

• Neuroscience
• Artificial Intelligence
• Developmental Psychology

Background:

• Multimodal neuroimaging offers comprehensive brain insights but faces integration challenges.
• Predicting cognitive outcomes in children is crucial for understanding neurodevelopment.

Purpose of the Study:

• To develop and evaluate a deep learning-based multimodal fusion framework for predicting cognitive outcomes in children.
• To compare different fusion strategies for integrating structural MRI (gray matter density) and diffusion MRI (white matter fractional anisotropy) data.

Main Methods:

• Utilized data from the Adolescent Brain Cognitive Development (ABCD) Study.
• Extracted modality-specific features using separate convolutional neural networks (CNNs).
• Integrated features via concatenation, multi-head attention, and transformer encoder-based fusion.

Main Results:

• Direct feature concatenation achieved the highest predictive performance (test correlation of 0.44).
• Multimodal models outperformed single-modality approaches, demonstrating the added value of data integration.
• Guided Grad-CAM identified key gray matter and white matter regions influencing intelligence prediction.

Conclusions:

• Deep learning-based multimodal fusion is effective for predicting cognitive outcomes in children.
• Simple feature concatenation is a robust strategy for integrating neuroimaging modalities.
• The framework provides interpretable insights into brain-region contributions to cognitive abilities.