Enhancing graph neural networks through universal self-knowledge distillation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Graph Neural Networks

Background:

• Graph self-distillation methods offer advantages over graph distillation, with reduced memory and time demands.
• Existing self-distillation techniques often rely on handcrafted soft labels via auxiliary branches or contrastive learning, which can still be computationally intensive.
• These methods, while faster than traditional graph knowledge distillation (KD), incur overhead compared to direct model training.

Purpose of the Study:

• To introduce a novel, general, and effective soft label acquisition method for graph self-distillation.
• To address the limitations of existing self-distillation approaches by reducing computational overhead.
• To enable models to distill knowledge from their own intermediate outputs efficiently.

Main Methods:

• Proposed Universal Graph Self-Knowledge Distillation (UGKD), a method for acquiring soft labels from a model's intermediate outputs.
• Utilized the student model's target logit as soft target labels.
• Generated soft non-target labels based on the ranking of intermediate features, guided by Zipf's law.

Main Results:

• UGKD is the first graph self-knowledge distillation method compatible with both MLP and GNN models.
• Demonstrated state-of-the-art performance with negligible increases in time and memory usage.
• Achieved significant improvements in student model performance through efficient self-distillation.

Conclusions:

• UGKD provides an effective and efficient solution for graph self-knowledge distillation.
• The method's low overhead makes it practical for various graph-based machine learning tasks.
• UGKD represents a significant advancement in self-distillation techniques for graph models.