Quantformer: Learning Extremely Low-Precision Vision Transformers.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Conventional network quantization methods struggle with transformer architectures, leading to significant self-attention deviation and quantization errors.
• Existing quantization strategies applied to weights and activations of fully-connected layers do not adequately address the unique properties of vision transformers.
• The diverse distribution of patch features in transformers complicates a unified quantization approach, causing severe accuracy loss.

Purpose of the Study:

• To propose Quantformer, an extremely low-precision vision transformer designed for efficient inference.
• To address the challenges of quantization in vision transformers, specifically self-attention deviation and patch feature errors.
• To develop novel quantization techniques that preserve self-attention rank and minimize quantization errors in patch features.

Main Methods:

• Enforcing self-attention rank consistency in quantized transformers to mimic full-precision counterparts using capacity-aware distribution.
• Implementing a group-wise discretization strategy for patch features to minimize quantization errors.
• Utilizing adaptive concentration degree, selected via self-attention entropy, to preserve rank consistency and adapt model capacity.
• Employing differentiable group assignment for partitioning patch features, enabling diverse discretization strategies and reducing rounding/clipping errors.

Main Results:

• Quantformer significantly outperforms state-of-the-art network quantization methods on image classification and object detection tasks.
• The proposed method demonstrates superior performance across various vision transformer architectures.
• Integration with mixed-precision quantization further enhances the performance of vanilla vision transformer models.

Conclusions:

• Quantformer offers an effective solution for efficient low-precision inference in vision transformers.
• The novel quantization strategies for self-attention and patch features successfully mitigate accuracy degradation.
• Quantformer represents a significant advancement in efficient deep learning model deployment for computer vision applications.