A Survey on Efficient Vision Transformers: Algorithms, Techniques, and Performance Benchmarking.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Vision Transformers (ViT) excel at global information extraction via self-attention, surpassing Convolutional Neural Networks.
• ViT performance scales with size, parameters, and operations, leading to high computational and memory demands.
• Quadratic increase in self-attention cost with image resolution challenges real-world deployment due to hardware limitations.

Purpose of the Study:

• To investigate efficient methodologies for Vision Transformer (ViT) architectures.
• To ensure sub-optimal estimation performances despite hardware and environmental restrictions.
• To analyze strategies for making ViTs suitable for real-world applications.

Main Methods:

• Analysis of four efficient categories: compact architecture, pruning, knowledge distillation, and quantization.
• Introduction of a new metric, Efficient Error Rate, for comparing models based on inference-time hardware impact.
• Mathematical definition and discussion of state-of-the-art efficient ViT methodologies.

Main Results:

• Detailed mathematical definitions of efficiency strategies for Vision Transformers.
• Comprehensive description and discussion of current state-of-the-art efficient methodologies.
• Performance analysis of these methodologies across various application scenarios.

Conclusions:

• Efficient methodologies are crucial for deploying Vision Transformers in resource-limited settings.
• The Efficient Error Rate metric provides a standardized way to evaluate model efficiency.
• Further research into open challenges and promising directions can advance efficient ViT development.