Fine-Grained Learning Behavior-Oriented Knowledge Distillation for Graph Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Graph Neural Networks

Background:

• Knowledge distillation (KD) is crucial for compressing graph neural networks (GNNs) for resource-constrained devices.
• The influence of knowledge complexity and inter-model learning behavior differences on GNN distillation efficiency is not well understood.

Purpose of the Study:

• To address underexplored aspects of GNN distillation, specifically knowledge complexity and learning behavior differences.
• To propose a novel KD method, fine-grained learning behavior (FLB), to improve GNN compression and deployment.

Main Methods:

• Feature knowledge decoupling (FKD): Separates student network features into teacher-related (TRFs) and downstream (DFs) for focused learning.
• Teacher learning behavior guidance (TLBG): Maps teacher model behaviors to correct student learning deviations.
• Implementation of FLB comprising both FKD and TLBG components.

Main Results:

• Extensive experiments conducted on eight datasets and 12 baseline frameworks.
• The proposed FLB method significantly improved the performance of student GNNs.
• Enhanced robustness of student GNNs was observed within their original frameworks.

Conclusions:

• The fine-grained learning behavior (FLB) method offers a significant advancement in GNN knowledge distillation.
• FLB effectively addresses knowledge complexity and learning behavior disparities, leading to more efficient GNN compression.
• The approach facilitates the deployment of more capable GNNs on resource-limited platforms.