Compressing Transfer: Mutual Learning- Empowered Knowledge Distillation for Temporal Knowledge Graph Reasoning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Science

Background:

• Temporal Knowledge Graph Reasoning (TKGR) models are increasingly used, creating a need for reduced memory consumption and improved efficiency.
• Knowledge Distillation (KD) is a technique for model compression and acceleration, now being applied to TKGR.
• Existing KD methods face challenges in extracting high-value knowledge and optimizing the teaching pattern for TKGR.

Purpose of the Study:

• To address the challenges in knowledge transfer for TKGR model compression.
• To develop a more effective knowledge distillation framework for TKGR models.
• To improve the efficiency and reduce the memory footprint of TKGR models.

Main Methods:

• A soft-label evaluation mechanism was designed to measure confidence and entropy changes, mitigating anomaly diffusion and knowledge transfer redundancy.
• A Mutual Learning-empowered KD (MLEMKD) framework was proposed for compressing TKGR models.
• The framework refines knowledge distribution by analyzing cognitive differences between teacher and student models.

Main Results:

• The proposed soft-label evaluation mechanism effectively mitigates anomaly diffusion and knowledge transfer redundancy.
• The MLEMKD framework enhances knowledge acceptability by refining knowledge distribution based on cognitive differences.
• Extensive experiments on four benchmark datasets show MLEMKD significantly outperforms existing KD methods.

Conclusions:

• The MLEMKD framework offers a superior approach to compressing TKGR models compared to existing methods.
• The study highlights the importance of optimizing knowledge transfer patterns in KD for TKGR.
• MLEMKD demonstrates significant improvements in efficiency and performance for TKGR models.