Intra-head pruning for vision transformers via inter-layer dimension relationship modeling.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Transformer models achieve high performance in NLP and computer vision but incur significant computational costs.
• Existing head pruning methods for transformers often lead to accuracy loss due to coarse granularity and ignoring inter-layer dependencies.
• Efficient compression of transformer networks requires addressing these limitations for practical deployment.

Purpose of the Study:

• To propose a novel intra-head pruning (IHP) technique for efficient sparse training of vision transformers.
• To develop a method that minimizes computational cost while preserving network accuracy.
• To overcome the limitations of existing pruning strategies in transformer models.

Main Methods:

• Introduced a trainable row parameter for sparse training within transformer heads.
• Developed a relationship matrix to guide a grouping pruning process for component elimination.
• Ensured consistent elimination of redundant components to maintain structural and functional integrity.

Main Results:

• Demonstrated significant reduction in computational cost for vision transformers like DeiT, Swin Transformer, and CCT on benchmark datasets (CIFAR-10/100, ImageNet-1K).
• Achieved minimal accuracy decrease across tested models.
• On ILSVRC-12, IHP improved Top-1 accuracy by 0.47% for DeiT-tiny compared to advanced methods at a 46.20% FLOPs reduction.

Conclusions:

• Intra-head pruning (IHP) offers an effective strategy for compressing vision transformers.
• The proposed method balances computational efficiency and accuracy preservation.
• IHP provides a viable solution for deploying transformer models on resource-constrained devices.