Progressive fine-to-coarse reconstruction for accurate low-bit post-training quantization in vision transformers

Area of Science:

• Computer Vision
• Deep Learning Model Compression

Background:

• Post-Training Quantization (PTQ) is efficient for compressing Vision Transformers (ViTs) but often causes performance degradation at low bit-widths.
• Existing reconstruction methods in PTQ use fixed granularity, failing to leverage progressive relationships for optimal ViT compression.

Purpose of the Study:

• To propose a novel Progressive Fine-to-Coarse Reconstruction (PFCR) method to improve the accuracy of low-bit quantized Vision Transformers.
• To introduce a Progressive Optimization Strategy (POS) to facilitate training and further enhance model performance.

Main Methods:

• PFCR reconstructs fine-grained units (multi-head self-attention, multi-layer perceptron modules, and shortcuts) and progressively combines them for coarser granularity reconstruction.
• A Progressive Optimization Strategy (POS) is employed to address training challenges associated with PFCR.

Main Results:

• The proposed PFCR method achieves state-of-the-art Top-1 accuracy on the ImageNet dataset for low-bit quantized ViTs.
• Specifically, 3-bit quantized ViT-B achieved 75.61% Top-1 accuracy using PTQ with PFCR.
• Evaluations on the COCO dataset demonstrate the method's effectiveness and generalization for object detection and instance segmentation.

Conclusions:

• PFCR offers a significant improvement in the accuracy of low-bit quantized Vision Transformers.
• The method demonstrates strong performance and generalization across various computer vision tasks.