Multiple interpretation ensemble distillation for graph neural networks

Existing graph knowledge distillation methods suffer from limited absorption of the teacher's "dark knowledge" because they rely on simple logit alignment, which often causes overfitting or incomplete capture of underlying patterns. Additionally, relying on a single perspective severely restricts the student's learning effectiveness and generalization ability. To address these issues, we develop a novel Multiple Interpretation Ensemble Distillation (MIED) method. It constructs a multi-interpreter composed of multiple single-layer MLPs for the student, termed the Student Interpretation (SI) component, to interpret knowledge from diversified outputs, thus avoiding representational bias from a single student output. Based on this, it introduces two effective strategies, i.e., Hybrid Sampling and Hierarchical Update. The former employs different sampling strategies for the outputs of the teacher and student (including the SI component). Specifically, the teacher's output adopts a percentage random sampler, while the outputs of the student and SI component both leverage a positive-negative sampler. With this design, MIED can facilitate better coordination of sample selection and the learning process among the teacher, student, and SI component. The latter updates the parameters of the last layer in the student using the exponential moving average of the fused parameters of the SI component, while the parameters of other layers are updated via a regular optimizer. This enhances the robustness and generalization performance of MIED. Extensive experiments on seven real-world public datasets demonstrate that MIED outperforms existing methods in node classification tasks, resulting in an average improvement of 5.56% over GCN and 27.43% over MLP, respectively. Moreover, compared with directly using multiple students (where the number is consistent with the number of layers in the SI component), MIED achieves improvements approximately 6.00% in time, 50.00% in space, and 0.20% in accuracy. These results indicate that MIED is scalable and generalizable, and exhibits robustness on complex samples.