ADFQ-ViT: Activation-Distribution-Friendly post-training Quantization for Vision Transformers.

Area of Science:

• Computer Vision
• Deep Learning
• Model Optimization

Background:

• Vision Transformers (ViTs) achieve high performance but require substantial computational resources.
• Existing quantization methods struggle with accuracy loss in low-bit scenarios for ViTs.
• Distinct activation distributions in ViTs challenge conventional quantization techniques.

Purpose of the Study:

• To develop a novel post-training quantization framework for Vision Transformers (ViTs) that minimizes accuracy loss at low bit-widths.
• To address the unique activation distribution characteristics within ViTs that hinder standard quantization approaches.
• To enable efficient inference of ViTs on resource-constrained devices.

Main Methods:

• Proposed Activation-Distribution-Friendly post-training Quantization for Vision Transformers (ADFQ-ViT).
• Introduced Per-Patch Outlier-aware Quantizer for post-LayerNorm activations.
• Designed Shift-Log2 Quantizer for non-uniform post-GELU activations.
• Implemented Attention-score enhanced Module-wise Optimization for error mitigation.

Main Results:

• ADFQ-ViT demonstrates significant improvements over baselines in image classification, object detection, and instance segmentation at 4-bit.
• Achieved a 5.17% increase in Top-1 accuracy on ImageNet for a 4-bit quantized ViT-B model.
• Outperformed existing methods in low-bit quantization accuracy for Vision Transformers.

Conclusions:

• ADFQ-ViT effectively handles unique activation distributions in ViTs for accurate low-bit quantization.
• The proposed framework enables efficient deployment of ViTs on resource-limited hardware without substantial performance degradation.
• ADFQ-ViT represents a significant advancement in optimizing Vision Transformers for practical applications.