Vicinity Vision Transformer.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Vision transformers (ViTs) excel in computer vision but struggle with high-resolution images due to quadratic complexity of softmax attention.
• Existing linear attention methods from NLP do not fully leverage the inherent 2D locality crucial for vision tasks.
• High computational and memory demands of traditional ViTs limit their scalability to larger image dimensions.

Purpose of the Study:

• To develop a linear attention mechanism for vision transformers that efficiently handles high-resolution images.
• To incorporate 2D locality bias into the attention mechanism to improve performance on vision tasks.
• To reduce the computational complexity and parameter count of vision transformers without sacrificing accuracy.

Main Methods:

• Proposed Vicinity Attention, a linear attention mechanism that weights patches based on 2D Manhattan distance, prioritizing local neighbors.
• Introduced the Vicinity Attention Block with Feature Reduction Attention (FRA) and Feature Preserving Connection (FPC) to manage feature dimension complexity.
• Developed the Vicinity Vision Transformer (VVT), a pyramid-structured backbone incorporating Vicinity Attention for general vision tasks.

Main Results:

• Vicinity Attention effectively integrates 2D locality into linear attention, achieving linear complexity.
• The Vicinity Attention Block reduces computation without accuracy degradation.
• The Vicinity Vision Transformer (VVT) demonstrates state-of-the-art image classification accuracy on CIFAR-100, ImageNet-1k, and ADE20K datasets.
• VVT exhibits a slower computational overhead growth rate with increasing input resolution compared to prior methods.
• VVT achieves state-of-the-art results with 50% fewer parameters than existing approaches.

Conclusions:

• Vicinity Attention offers an effective solution for scaling vision transformers to high-resolution images by incorporating 2D locality.
• The Vicinity Vision Transformer (VVT) provides a computationally efficient and parameter-light alternative for various computer vision tasks.
• This work presents a significant advancement in making transformer-based models more practical and performant for real-world, high-resolution vision applications.