On Representation Knowledge Distillation for Graph Neural Networks.

Area of Science:

• Machine Learning
• Graph Neural Networks
• Artificial Intelligence

Background:

• Knowledge distillation (KD) trains efficient student Graph Neural Networks (GNNs) using larger teacher models.
• Existing methods like local structure preserving (LSP) loss focus on local graph structures.
• Real-world graphs possess complex global topology, including latent interactions and noise, which current methods may not fully capture.

Purpose of the Study:

• To investigate if preserving global topology is a more effective distillation objective for GNNs.
• To introduce a novel distillation method that captures global graph structure.
• To evaluate the proposed method on large-scale, real-world graph datasets.

Main Methods:

• Propose graph contrastive representation distillation (G-CRD), a novel KD technique.
• Utilize contrastive learning to align student and teacher node embeddings in a shared space, implicitly preserving global topology.
• Benchmark G-CRD on extensive datasets and diverse GNN architectures.

Main Results:

• G-CRD consistently improves the performance and robustness of lightweight GNNs across various datasets.
• The proposed G-CRD method outperforms LSP and its variants, as well as 2D computer vision baselines.
• Analysis shows G-CRD effectively balances the preservation of both local and global graph relationships.

Conclusions:

• G-CRD offers a superior approach to knowledge distillation for GNNs by focusing on global topology preservation.
• The method enhances the capabilities of resource-efficient GNNs, making them more effective on complex graph data.
• G-CRD provides a balanced representation learning strategy, outperforming methods focused solely on local or global structures.