MMF-NNs: Multi-modal Multi-granularity Fusion Neural Networks for brain networks and its application to epilepsy...

Area of Science:

• Neuroimaging
• Artificial Intelligence
• Medical Diagnostics

Background:

• Structural and functional brain networks from MRI offer complementary insights into brain diseases.
• Current fusion models often overlook the multi-granularity nature of brain networks (edge, node, graph levels).
• Effective fusion of multi-modal brain network data is crucial for improved disease identification.

Purpose of the Study:

• To propose a novel Multi-modal Multi-granularity Fusion Neural Networks (MMF-NNs) framework.
• To integrate multi-modal brain network features across global (graph-level) and local (edge-level, node-level) granularities.
• To leverage the full topological information for enhanced brain disease identification.

Main Methods:

• Developed an interactive feature learning module for edge-level and node-level feature maps of structural and functional brain networks.
• Designed a multi-modal decomposition bilinear pooling module for graph-level joint representation learning.
• Implemented a Multi-modal Multi-granularity Fusion Neural Networks (MMF-NNs) framework.

Main Results:

• The MMF-NNs framework effectively fuses multi-modal brain network information at various granularities.
• Experiments on epilepsy datasets showed superior performance compared to existing state-of-the-art methods.
• The proposed framework successfully utilizes both local and global topological information.

Conclusions:

• The MMF-NNs framework offers a significant advancement in brain disease identification by incorporating multi-granularity fusion.
• This approach enhances the utilization of topological information from multi-modal brain networks.
• MMF-NNs demonstrate high potential for clinical applications in diagnosing neurological disorders like epilepsy.