GRLT: Learning more from teachers by rethinking knowledge distillation from GNNs to MLPs

Area of Science:

• Machine Learning
• Graph Neural Networks
• Artificial Intelligence

Background:

• Graph Neural Networks (GNNs) are effective but have high inference latency.
• GNN-to-MLP (G2M) distillation reduces latency but struggles with node importance and cross-architecture transferability.
• Existing G2M methods neglect node significance and the model gap between GNN teachers and MLP students.

Purpose of the Study:

• To propose a novel G2M distillation method, Hybrid Graph Representation and Latent Space Translation (GRLT), to address limitations in current approaches.
• To enhance the performance of student MLPs by improving graph representation and knowledge transferability.
• To capture richer graph information and enable student models to focus on significant nodes.

Main Methods:

• Incorporating graph prior knowledge into a hybrid graph representation module for structural embeddings and aggregated node features.
• Implementing a latent space translation feature distillation module using Generalized Procrustes Analysis (GPA) for representation alignment.
• Utilizing cosine similarity for feature distillation in a two-stage process to bridge the model gap.

Main Results:

• GRLT demonstrated superior overall performance across seven experimental datasets.
• Distilled student MLPs using GRLT showed significant average performance improvements.
• Specifically, with SAGE as the teacher, GRLT achieved 1.72% improvement over NOSMOG, 3.72% over the teacher GNN, and 22.41% over vanilla MLP.

Conclusions:

• GRLT effectively enhances GNN-to-MLP distillation by improving graph representation and knowledge transferability.
• The proposed method successfully addresses the neglect of node importance and the model gap in existing G2M techniques.
• GRLT offers a promising approach for developing efficient and high-performing student MLPs from complex GNN models.