Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study

Area of Science:

• Artificial Intelligence
• Machine Learning
• Graph Neural Networks

Background:

• Training deep graph neural networks (GNNs) faces challenges including vanishing gradients, overfitting, and over-smoothing.
• Existing methods to improve deep GNN training lack standardized evaluation, making it hard to assess their true impact.
• A fair and reproducible benchmark is needed to compare different training techniques for deep GNNs.

Purpose of the Study:

• To establish the first fair and reproducible benchmark for assessing training techniques ('tricks') in deep GNNs.
• To systematically evaluate and categorize existing methods for training deep GNNs.
• To identify optimal combinations of techniques for achieving state-of-the-art performance in deep GNNs.

Main Methods:

• Developed a standardized benchmark for evaluating deep GNN training techniques.
• Categorized existing approaches and investigated their hyperparameter sensitivity.
• Conducted comprehensive evaluations on diverse graph datasets, including the Open Graph Benchmark, using various deep GNN backbones.

Main Results:

• Demonstrated that a combination of initial connection, identity mapping, group normalization, and batch normalization achieves new state-of-the-art results for deep GNNs.
• Identified specific training techniques that are crucial for enhancing deep GNN performance on large-scale graphs.
• Provided a unified configuration for consistent experimental settings.

Conclusions:

• The proposed benchmark enables fair and reproducible assessment of deep GNN training strategies.
• An integrated approach combining specific architectural and normalization techniques significantly boosts deep GNN performance.
• This work facilitates future research by providing a reliable platform for developing and validating advanced deep GNN training methods.