I&S-ViT: An Inclusive & Stable Method for Post-Training ViTs Quantization.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Vision Transformers (ViTs) offer scalable performance but have high computational costs, limiting industrial adoption.
• Post-training quantization (PTQ) reduces ViT costs but often leads to performance degradation, especially at lower bit-widths.

Purpose of the Study:

• To develop a novel method, I&S-ViT, for inclusive and stable post-training quantization of Vision Transformers.
• To address quantization inefficiencies in post-Softmax activations and the rugged loss landscape in post-LayerNorm activations during PTQ.

Main Methods:

• Introduced a shift-uniform-log2 quantizer (SULQ) for improved domain representation and distribution approximation of post-Softmax activations.
• Developed a three-stage smooth optimization strategy (SOS) combining channel-wise and layer-wise quantization for stable learning of post-LayerNorm activations.

Main Results:

• I&S-ViT demonstrates superior performance over existing PTQ methods for ViTs, particularly in low-bit quantization scenarios.
• Achieved a significant performance improvement of 50.68% for W3A3 ViT-B, showcasing the method's effectiveness.

Conclusions:

• I&S-ViT effectively mitigates performance loss in low-bit quantized ViTs, enhancing their viability for industrial applications.
• The proposed SULQ and SOS components offer a robust approach to inclusive and stable PTQ for Vision Transformers.