Decoupled graph knowledge distillation: A general logits-based method for learning MLPs on graphs.

Area of Science:

• Machine Learning
• Graph Neural Networks
• Artificial Intelligence

Background:

• Graph Neural Networks (GNNs) excel at non-Euclidean data but are computationally intensive for real-time applications.
• Graph Knowledge Distillation (KD) trains efficient Multi-layer Perceptrons (MLPs) to replace GNNs.
• Current KD methods often overlook logit layer distillation, focusing on intermediate features.

Purpose of the Study:

• To introduce a novel logits-based graph knowledge distillation method.
• To address the limitations of existing graph KD approaches by focusing on the logit layer.
• To improve the efficiency and accuracy of student MLPs in graph-based tasks.

Main Methods:

• Introduced Decoupled Graph Knowledge Distillation (DGKD) focusing on logit layer distillation.
• Reformulated KD loss into Target Class Graph Distillation (TCGD) and Non-Target Class Graph Distillation (NCGD) losses.
• Decoupled the negative correlation between prediction confidence and NCGD, and eliminated fixed weights between TCGD and NCGD.

Main Results:

• DGKD demonstrated improved prediction accuracy for student MLPs across benchmark datasets.
• The method achieved superior performance compared to existing graph KD techniques.
• DGKD showed flexibility as a plug-and-play component for enhancing other KD frameworks.

Conclusions:

• DGKD offers an effective logits-based approach to graph knowledge distillation.
• The decoupling strategy enhances student MLP performance by optimizing distillation loss components.
• DGKD provides a valuable and adaptable tool for deploying efficient GNN-like models in industrial settings.