Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Graph Neural Networks (GNNs) are powerful deep learning tools for complex network problems.
• GNNs face challenges in efficient execution on parallel architectures due to their unique computational demands.

Purpose of the Study:

• To design a taxonomy for GNN parallelism (data, model, pipelining).
• To analyze parallelism in GNN models, tasks, software, and hardware.
• To provide insights for maximizing GNN performance and identify future research directions.

Main Methods:

• Developed a taxonomy of parallelism in GNNs.
• Utilized the work-depth model to assess parallelism, communication, and synchronization.
• Analyzed tensor sparsity/density for vectorization.
• Formally analyzed GNN pipelining and generalized Message-Passing models.
• Investigated asynchronous parallel GNN pipelines.

Main Results:

• Quantified parallelism across various GNN models and frameworks.
• Identified key factors influencing GNN performance on parallel architectures.
• Generalized GNN pipelining for deeper optimizations.
• Explored asynchronicity for future parallel GNNs.

Conclusions:

• The taxonomy provides a framework for understanding and optimizing GNN parallelism.
• Insights derived can significantly enhance GNN computational efficiency.
• The work lays the foundation for advanced asynchronous parallel GNN designs.