Multiple interpretation ensemble distillation for graph neural networks

Area of Science:

• Artificial Intelligence
• Machine Learning
• Graph Neural Networks

Background:

• Existing graph knowledge distillation methods struggle with limited "dark knowledge" absorption due to simple logit alignment, leading to overfitting and incomplete pattern capture.
• A single student perspective restricts learning effectiveness and generalization ability in graph-based tasks.

Purpose of the Study:

• To introduce a novel Multiple Interpretation Ensemble Distillation (MIED) method for improved graph knowledge distillation.
• To address limitations of existing methods by enabling diversified knowledge interpretation and enhancing student model robustness and generalization.

Main Methods:

• Developed the Student Interpretation (SI) component, a multi-interpreter using multiple single-layer MLPs, to interpret knowledge from diversified student outputs, mitigating representational bias.
• Introduced Hybrid Sampling with different strategies for teacher (percentage random) and student/SI component (positive-negative) outputs to coordinate sample selection.
• Implemented Hierarchical Update to enhance robustness and generalization by using exponential moving average for the student's last layer parameters based on SI component fusion.

Main Results:

• MIED significantly outperforms existing methods in node classification tasks on seven real-world datasets, showing average improvements of 5.56% over Graph Convolutional Networks (GCN) and 27.43% over Multi-Layer Perceptrons (MLP).
• Compared to using multiple individual students, MIED achieves comparable or better accuracy with significant improvements in efficiency (6.00% faster, 50.00% less space).

Conclusions:

• MIED offers a scalable, generalizable, and robust solution for graph knowledge distillation, particularly effective on complex samples.
• The proposed method successfully enhances the absorption of teacher "dark knowledge" and improves student model performance beyond traditional approaches.