Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Vision Transformers (ViTs) rely on a subset of informative regions for accurate image recognition.
• Existing acceleration methods often lack adaptivity to input-specific visual data characteristics.

Purpose of the Study:

• To develop a dynamic token sparsification framework for accelerating vision Transformer models.
• To propose a generalizable approach for accelerating various deep learning architectures using adaptive computation.

Main Methods:

• A dynamic token sparsification framework that progressively prunes redundant tokens based on input.
• A lightweight prediction module integrated into different layers to estimate token importance hierarchically.
• A generic dynamic spatial sparsification framework with progressive sparsification and asymmetric computation for structured feature maps.

Main Results:

• Hierarchically pruning 66% of input tokens reduced computations by 31%-35% and improved throughput by over 40% with <0.5% accuracy drop in ViTs.
• Similar acceleration achieved on Convolutional Neural Networks (CNNs) and Swin Transformers using asymmetric computation.
• Promising results demonstrated on complex tasks like semantic segmentation and object detection.

Conclusions:

• Dynamic spatial sparsification is an effective approach for accelerating deep learning models.
• Adaptive and asymmetric computation offers a general solution for enhancing model efficiency across diverse architectures.
• The proposed framework significantly reduces computational load while maintaining high accuracy, paving the way for more efficient visual data processing.