Magnitude Pruning of Large Pretrained Transformer Models with a Mixture Gaussian Prior.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Large pretrained transformer models achieve state-of-the-art performance in AI applications.
• Substantial parameter counts in these models present deployment challenges.
• Existing pruning methods like magnitude pruning have limitations, especially for transfer learning in NLP.

Purpose of the Study:

• Introduce a novel magnitude-based pruning algorithm, Mixture Gaussian Prior Pruning (MGPP).
• Address the challenge of deploying large AI models by reducing their size.
• Retain model expressiveness during pruning.

Main Methods:

• Developed MGPP, a magnitude-based pruning algorithm utilizing a mixture Gaussian prior for regularization.
• Pruned non-expressive weights guided by the mixture Gaussian prior.
• Conducted extensive evaluations across diverse NLP tasks.

Main Results:

• MGPP demonstrated superiority over existing pruning methods, especially in high sparsity settings.
• Evaluations included natural language understanding, question answering, and natural language generation tasks.
• The algorithm effectively retains the model's expressive capability.

Conclusions:

• MGPP offers an effective solution for compressing large transformer models.
• The proposed method enhances performance in various NLP tasks.
• Theoretical justification for the sparse transformer's consistency supports MGPP's effectiveness.