Pea-KD: Parameter-efficient and accurate Knowledge Distillation on BERT.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Knowledge Distillation (KD) is a model compression technique training smaller student models from larger teacher models.
• Existing KD methods face limitations due to inherent student model capacity constraints and lack of effective initial guidance.
• These limitations result in suboptimal performance for conventional KD approaches.

Purpose of the Study:

• To introduce Pea-KD (Parameter-efficient and accurate Knowledge Distillation), a novel KD approach addressing current limitations.
• To enhance student model capacity and provide a specialized initialization strategy for improved imitation of teacher models.
• To achieve significant performance gains in model compression tasks.

Main Methods:

• Pea-KD incorporates Shuffled Parameter Sharing (SPS) to increase student model capacity.
• Pea-KD utilizes Pretraining with Teacher's Predictions (PTP) as a specialized initialization method for student models.
• The combination of SPS and PTP aims to alleviate the inherent limitations of traditional KD.

Main Results:

• Experiments on BERT across various datasets and tasks demonstrate significant performance improvements.
• The proposed Pea-KD approach achieved an average improvement of 4.4% on four GLUE tasks.
• Pea-KD outperformed existing KD baselines by substantial margins.

Conclusions:

• Pea-KD effectively addresses the limitations of conventional Knowledge Distillation.
• The combination of SPS and PTP offers a powerful strategy for parameter-efficient and accurate model compression.
• Pea-KD represents a significant advancement in improving student model performance through enhanced distillation techniques.