Transformer based multi-modal MRI fusion for prediction of post-menstrual age and neonatal brain development analysis

The brain development during the perinatal period is characterized by rapid changes in both structure and function, which have significant impact on the cognitive and behavioral abilities later in life. Accurate assessment of brain age is a crucial indicator for brain development maturity and can help predict the risk of neonatal pathology. However, evaluating neonatal brains using magnetic resonance imaging (MRI) is challenging due to its complexity, multi-dimension, and noise with subtle alterations. In this paper, we propose a multi-modal deep learning framework based on transformers for precise post-menstrual age (PMA) estimation and brain development analysis using T2-weighted structural MRI (T2-sMRI) and diffusion MRI (dMRI) data. First, we build a two-stream dense network to learn modality-specific features from T2-sMRI and dMRI of brain individually. Then, a transformer module based on self-attention mechanism integrates these features for PMA prediction and preterm/term classification. Finally, saliency maps on brain templates are used to enhance the interpretability of results. Our method is evaluated on the multi-modal MRI dataset of the developing Human Connectome Project (dHCP), which contains 592 neonates, including 478 term-born and 114 preterm-born subjects. The results demonstrate that our method achieves a 0.5-week mean absolute error (MAE) in PMA estimation for term-born subjects. Notably, preterm-born subjects exhibit delayed brain development, worsening with increasing prematurity. Our method also achieves 95% accuracy in classification of term-born and preterm-born subjects, revealing significant group differences.