DDK: Dynamic structure pruning based on differentiable search and recursive knowledge distillation for BERT.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Large-scale pre-trained models (e.g., BERT) excel in NLP but require substantial computational resources.
• High parameter counts hinder practical deployment due to storage and processing demands.

Purpose of the Study:

• To propose a novel compression technique for large pre-trained language models.
• To enhance the efficiency and deployability of models like BERT.

Main Methods:

• Introduced DDK, a method combining dynamic structure pruning and recursive knowledge distillation.
• Utilized differentiable search to optimize feed-forward layer channels and self-attention heads.
• Implemented recursive knowledge distillation with adaptive weighting for feature extraction from intermediate layers.

Main Results:

• The DDK method successfully compressed and accelerated BERT models.
• Experimental results on the GLUE benchmark demonstrated superior performance compared to existing methods.
• Ablation analysis validated the effectiveness of the proposed pruning and distillation strategies.

Conclusions:

• The DDK method offers an effective solution for compressing large pre-trained language models.
• This approach addresses the practical deployment challenges of resource-intensive NLP models.
• DDK achieves state-of-the-art performance while significantly improving model efficiency.