Blockwise compression of transformer-based models without retraining.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Transformer models (e.g., GPT-3, ChatGPT, GPT-4) excel at language tasks but demand substantial computational resources and memory.
• Current compression techniques like layerwise quantization often degrade model accuracy and require extensive retraining.

Purpose of the Study:

• To introduce Blockwise Compression for Transformers (BCT), a retraining-free framework for efficient transformer model deployment.
• To address the computational and memory challenges associated with large transformer models.

Main Methods:

• BCT employs blockwise compression across the entire transformer architecture, including embeddings, matrix multiplications, and other components.
• This method mitigates data distribution shifts caused by quantization, thus eliminating the need for retraining.

Main Results:

• A case study demonstrated up to 7.988x compression on an efficient model using BCT.
• Evaluation on General Language Understanding Evaluation (GLUE) datasets showed less than 0.9% accuracy degradation, outperforming other methods.

Conclusions:

• BCT provides an effective solution for compressing transformer models without accuracy loss or retraining.
• The framework facilitates easier deployment of large language models by significantly reducing their resource requirements.