Pruning Self-Attentions Into Convolutional Layers in Single Path.

Area of Science:

• Computer Vision
• Deep Learning
• Model Compression

Background:

• Vision Transformers (ViTs) excel in computer vision but suffer from high computational demands and limited local pattern modeling.
• Multi-head self-attention (MSA) layers in ViTs contribute to these challenges.
• Existing compression methods often struggle to balance efficiency and performance.

Purpose of the Study:

• To develop an efficient and automatic method for compressing pre-trained ViTs.
• To address the computational and inductive bias issues in ViTs.
• To introduce locality into compact ViT models.

Main Methods:

• Introduced Single-Path Vision Transformer pruning (SPViT), a novel weight-sharing scheme between MSA and convolutional operations.
• Utilized learnable binary gates to select operations within MSA layers and control Feed-Forward Network (FFN) layer expansion ratios.
• Created a unified search space for automated pruning and optimization of ViT architectures.

Main Results:

• SPViT achieved state-of-the-art (SOTA) results on ImageNet-1k for model pruning.
• Demonstrated significant FLOPs reduction (e.g., 52.0% for DeiT-B) while simultaneously improving top-1 accuracy by 0.6%.
• Successfully compressed ViTs, adding proper locality and reducing computational complexity.

Conclusions:

• SPViT offers an effective solution for compressing Vision Transformers, enhancing efficiency and performance.
• The proposed method provides a flexible and automated approach to prune ViTs.
• SPViT sets a new benchmark for pruning Vision Transformers, balancing computational reduction with accuracy gains.