Selecting and Distilling Cross-Label Models

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Knowledge distillation, using pre-trained 'teacher' models to improve 'student' models, is increasingly popular.
• A large number of pre-trained models necessitates efficient resource utilization strategies.
• Existing methods face challenges in selecting optimal models and handling diverse label spaces.

Purpose of the Study:

• To develop a universal model reuse approach for knowledge distillation.
• To address the impracticality of exhaustive pre-trained model testing.
• To overcome the semantic gap caused by differing label spaces in pre-trained models.

Main Methods:

• Introduced a dual-phase framework: Selective Cross-Label Distillation.
• Phase 1 (Model Assessment): Evaluated semantic similarity using optimal transport and transportation cost.
• Phase 2 (Knowledge Reuse): Minimized transportation cost between selected source and target models.

Main Results:

• The framework enables efficient selection of advantageous pre-trained models.
• Successfully bridges the semantic gap between diverse pre-trained models and target tasks.
• Experimental validation confirmed the framework's effectiveness in model selection and knowledge reuse.

Conclusions:

• Selective Cross-Label Distillation offers an effective solution for pre-trained model selection in knowledge distillation.
• The approach facilitates universal model reuse, regardless of label space differences.
• This framework enhances student model performance by optimizing the use of pre-trained resources.