The Multiscale Surface Vision Transformer.

Area of Science:

• Neuroimaging
• Computer Vision
• Machine Learning

Background:

• Surface meshes are crucial for representing human cortical data but pose challenges for deep learning due to complex topology and geometry.
• The quadratic cost of self-attention in Transformers limits their application in dense prediction tasks on such data.

Purpose of the Study:

• To introduce the Multiscale Surface Vision Transformer (MS-SiT), a novel backbone architecture for deep learning on cortical surface data.
• To address the computational challenges of Transformers for high-resolution surface analysis.

Main Methods:

• Developed MS-SiT, applying self-attention within local mesh windows and using a shifted-window strategy for improved information sharing.
• Implemented hierarchical representation learning by successively merging neighboring patches.
• Integrated MS-SiT into a U-shaped architecture for surface segmentation tasks.

Main Results:

• MS-SiT demonstrated superior performance in neonatal phenotyping prediction on the Developing Human Connectome Project (dHCP) dataset.
• Achieved competitive results in cortical parcellation on the UK Biobank (UKB) and MindBoggle datasets when integrated into a U-Net architecture.
• Code and models are publicly available.

Conclusions:

• MS-SiT offers an effective and efficient deep learning backbone for complex surface data analysis.
• The hierarchical approach enables learning representations suitable for diverse cortical surface prediction tasks.
• MS-SiT represents a significant advancement for neuroimaging and computational neuroscience applications.