CAIT: Triple-Win Compression Toward High Accuracy, Fast Inference, and Favorable Transferability for ViTs.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Vision Transformers (ViTs) achieve state-of-the-art performance but have high computational demands.
• Existing compression methods for ViTs offer suboptimal trade-offs between performance and speed.
• Current methods struggle with transferability to tasks requiring spatial image structure.

Purpose of the Study:

• To develop a joint compression method for ViTs that enhances accuracy, inference speed, and transferability.
• To address the limitations of existing token and channel pruning techniques.
• To enable efficient deployment of ViTs on resource-constrained devices.

Main Methods:

• Introduced an asymmetric token merging (ATME) strategy to integrate neighboring tokens while preserving spatial information.
• Developed a consistent dynamic channel pruning (CDCP) strategy to uniformly prune channels in multi-head self-attention modules.
• Implemented a joint compression approach combining ATME and CDCP for Vision Transformers.

Main Results:

• The proposed CAIT method achieves a superior balance between model accuracy and inference speed.
• Demonstrated effective compression of redundant token and channel information in ViTs.
• Achieved state-of-the-art performance across various ViTs on multiple benchmark datasets.

Conclusions:

• CAIT offers a highly effective joint compression strategy for Vision Transformers.
• The method significantly improves inference speed and accuracy without compromising spatial structure preservation.
• CAIT enhances the transferability of compressed ViTs to downstream vision tasks.