Revealing Continuous Brain Dynamical Organization with Multimodal Graph Transformer.

Area of Science:

• Neuroscience
• Computational Neuroscience
• Data Science

Background:

• Brain large-scale dynamics are influenced by structural connectivity (SC).
• Existing multimodal methods often simplify spatial or temporal brain data, missing complex dynamics.
• Understanding how spatio-temporal dynamics adapt to heterogeneous SC is crucial.

Purpose of the Study:

• To propose a novel spatio-temporal graph Transformer model for integrating structural and functional brain connectivity.
• To capture complex spatio-temporal dynamics by considering both spatial and temporal domains.
• To improve the understanding of structure-function interactions and their relation to behavior.

Main Methods:

• Developed a spatio-temporal graph Transformer model integrating multimodal brain data (fMRI, MRI, MEG).
• Employed contrastive learning and multi-head attention for heterogeneous node and graph representation.
• Incorporated T1-to-T2-weighted (T1w/T2w) imaging to map heterogeneity and enhance structure-function fit.

Main Results:

• The model successfully highlighted local properties of large-scale brain spatio-temporal dynamics.
• Demonstrated the ability to capture the dependence strength between functional connectivity and behavioral performance.
• Showcased improved model fit to structure-function interactions using the heterogeneity map.

Conclusions:

• The proposed method enables a more comprehensive explanation of complex brain dynamics across different modalities.
• This approach advances the analysis of multimodal brain data, respecting both spatial and temporal complexities.
• Offers a new framework for investigating the relationship between brain structure, function, and behavior.