Joint Dual Feature Distillation and Gradient Progressive Pruning for BERT compression.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Large pre-trained language models (PLMs) are computationally intensive.
• Model compression techniques like pruning and distillation are crucial for efficient deployment.
• Existing methods have limitations in optimality, bias, and knowledge transfer efficiency.

Purpose of the Study:

• To propose a novel joint pruning and distillation method for automatic compression of PLMs.
• To address the suboptimality and bias in traditional pruning methods.
• To improve knowledge transfer in distillation by better utilizing teacher model information.

Main Methods:

• Gradient Progressive Pruning (GPP): Achieves smooth convergence of unit importance to zero for higher sparsity.
• Dual Feature Distillation (DFD): Employs adaptive global teacher and local student feature fusion for enhanced knowledge transfer.
• Joint application of GPP and DFD for comprehensive PLM compression.

Main Results:

• GPP enables smoother pruning and supports higher sparsity levels compared to traditional methods.
• DFD facilitates a "preview-review" mechanism for effective multi-level knowledge extraction.
• The combined approach significantly outperforms state-of-the-art methods on the GLUE benchmark.

Conclusions:

• The proposed joint pruning and distillation method offers a superior approach to compressing PLMs.
• GPP and DFD effectively mitigate limitations of individual pruning and distillation techniques.
• This work advances efficient deployment of large language models without compromising performance.