SensiMix: Sensitivity-Aware 8-bit index & 1-bit value mixed precision quantization for BERT compression.

Area of Science:

• Natural Language Processing (NLP)
• Machine Learning
• Model Compression

Background:

• Pre-trained language models like BERT are powerful but computationally expensive.
• Large memory footprint and long inference times hinder BERT's practical application.
• Efficient model compression is crucial for deploying NLP models on resource-constrained devices.

Purpose of the Study:

• To develop a novel BERT compression method that reduces model size and inference time.
• To maintain the accuracy of BERT models after compression.
• To address the trade-off between model efficiency and performance.

Main Methods:

• Proposed SensiMix (Sensitivity-Aware Mixed Precision Quantization) for BERT compression.
• Applied 8-bit index quantization to sensitive BERT modules and 1-bit value quantization to insensitive modules.
• Introduced three novel 1-bit training techniques: Absolute Binary Weight Regularization, Prioritized Training, and Inverse Layer-wise Fine-tuning.
• Utilized FP16 and XNOR-Count General Matrix Multiplication (GEMM) for optimized inference.

Main Results:

• SensiMix achieved significant model compression, reducing size by 8x.
• Inference time was reduced by approximately 80%.
• Accuracy was maintained across four GLUE downstream tasks without noticeable drops.

Conclusions:

• SensiMix offers an effective approach to compress BERT models, balancing efficiency and accuracy.
• The sensitivity-aware quantization strategy and novel training methods are key to minimizing performance degradation.
• This method enables the deployment of lightweight and fast BERT models for practical NLP applications.