Compressing deep graph convolution network with multi-staged knowledge distillation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Graph Neural Networks

Background:

• Compressing deep graph convolution networks (GCNs) is crucial for deployment on resource-limited devices.
• Existing GCN compression methods often neglect the multi-hop aggregation inherent in deep GCN architectures.
• Effective compression requires preserving the core functionality of multi-hop feature aggregation.

Purpose of the Study:

• To propose a novel approach for compressing deep GCNs into compact, single-layered GCNs.
• To effectively distill knowledge from multiple GCN layers, including aggregation and task prediction.
• To maintain the multi-hop feature aggregation capability within a single GCN layer.

Main Methods:

• Introduced MustaD (Multi-staged knowledge Distillation), a novel multi-staged knowledge distillation technique.
• Distilled knowledge encompassing both multi-layer aggregation patterns and final task predictions.
• Designed to preserve the essential multi-hop feature aggregation mechanism of deep GCNs.

Main Results:

• MustaD successfully compresses deep GCNs to single-layered GCNs while preserving key functionalities.
• Achieved state-of-the-art performance compared to existing knowledge distillation-based compression methods.
• Demonstrated significant accuracy improvements, up to 4.21 percentage points, over the second-best methods on four real-world datasets.

Conclusions:

• MustaD offers an effective solution for compressing deep GCNs, enabling deployment on edge devices.
• The proposed multi-staged knowledge distillation effectively captures and transfers complex GCN behaviors.
• This approach represents a significant advancement in efficient GCN model compression.