ShiftKD: Benchmarking knowledge distillation under distribution shift.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Model Compression

Background:

• Knowledge Distillation (KD) successfully transfers knowledge from large to small models.
• The reliability of KD methods under distribution shift is underexplored.
• Distribution shift, where training and testing data distributions differ, can degrade KD performance.

Purpose of the Study:

• To propose a unified framework, ShiftKD, for benchmarking KD methods against distribution shifts.
• To systematically evaluate KD performance under diversity and correlation shifts.
• To identify key factors influencing student model training in KD.

Main Methods:

• Developed ShiftKD, a comprehensive evaluation benchmark.
• Included over 30 KD methods across algorithmic, data-driven, and optimization approaches.
• Utilized five benchmark datasets to assess performance under distribution shift.

Main Results:

• Conducted extensive experiments to reveal strengths and limitations of state-of-the-art KD methods.
• Analyzed the impact of data augmentation, pruning, optimizers, and evaluation metrics on student model training.
• Identified critical factors for robust KD performance.

Conclusions:

• ShiftKD provides an effective benchmark for assessing KD reliability in real-world scenarios.
• The findings will drive the development of more robust KD methods resilient to distribution shifts.
• This work facilitates advancements in model compression and deployment.