MCBERT: A multi-modal framework for the diagnosis of autism spectrum disorder

Within the domain of neurodevelopmental disorders, autism spectrum disorder (ASD) emerges as a distinctive neurological condition characterized by multifaceted challenges. The delayed identification of ASD poses a considerable hurdle in effectively managing its impact and mitigating its severity. Addressing these complexities requires a nuanced understanding of data modalities and the underlying patterns. Existing studies have focused on a single data modality for ASD diagnosis. Recently, there has been a significant shift towards multimodal architectures with deep learning strategies due to their ability to handle and incorporate complex data modalities. In this paper, we developed a novel multimodal ASD diagnosis architecture, referred to as Multi-Head CNN with BERT (MCBERT), which integrates bidirectional encoder representations from transformers (BERT) for meta-features and a multi-head convolutional neural network (MCNN) for the brain image modality. The MCNN incorporates two attention mechanisms to capture spatial (SAC) and channel (CAC) features. The outputs of BERT and MCNN are then fused and processed through a classification module to generate the final diagnosis. We employed the ABIDE-I dataset, a multimodal dataset, and conducted a leave-one-site-out classification to assess the model's effectiveness comprehensively. Experimental simulations demonstrate that the proposed architecture achieves a high accuracy of 93.4 %. Furthermore, the exploration of functional MRI data may provide a deeper understanding of the underlying characteristics of ASD.