Learning to Explore Distillability and Sparsability: A Joint Framework for Model Compression

Area of Science:

• Artificial Intelligence
• Computer Science

Background:

• Deep learning models achieve high performance but require significant computational resources.
• Model compression techniques like knowledge distillation and filter pruning reduce computational load.
• Existing methods often address either knowledge distillation or filter pruning, but not both simultaneously.

Purpose of the Study:

• To introduce a novel framework for model compression that integrates both knowledge distillation and filter pruning.
• To define and utilize model attributes 'distillability' and 'sparsability' for effective compression.
• To improve both accuracy and reduce model size in deep learning models.

Main Methods:

• A dynamically distillability-and-sparsability learning framework (DDSL) was developed.
• DDSL employs a teacher-student-dean architecture for guided knowledge transfer and dynamic supervision.
• An alternating direction method of multipliers (ADMM)-based joint optimization algorithm (KDP) was used for training.

Main Results:

• The proposed DDSL framework demonstrated superior performance.
• DDSL outperformed 24 existing state-of-the-art methods in model compression.
• The framework effectively balances model accuracy and size reduction.

Conclusions:

• The DDSL framework offers an effective approach to simultaneous knowledge distillation and filter pruning.
• The defined attributes of distillability and sparsability provide valuable guidance for model compression.
• This integrated approach significantly advances the field of efficient deep learning model design.