UniFormer: Unifying Convolution and Self-Attention for Visual Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Convolutional Neural Networks (CNNs) excel at local feature extraction but struggle with global dependencies.
• Vision Transformers (ViTs) capture long-range dependencies but can be redundant.
• Existing models face challenges in balancing local redundancy and global dependency in visual data.

Purpose of the Study:

• To propose a novel Unified Transformer (UniFormer) that integrates the strengths of both CNNs and ViTs.
• To develop an efficient and effective framework for visual representation learning.
• To address the limitations of existing models in capturing both local and global information.

Main Methods:

• UniFormer employs a transformer architecture with relation aggregators in shallow and deep layers to capture local and global token affinity.
• The model seamlessly integrates convolution and self-attention mechanisms.
• A concise hourglass design with token shrinking and recovering is utilized for efficiency.

Main Results:

• UniFormer achieved 86.3% top-1 accuracy on ImageNet-1K classification without extra training data.
• State-of-the-art performance was obtained on various downstream tasks including video classification (Kinetics-400/600, Something-Something V1/V2), object detection (COCO), semantic segmentation (ADE20K), and pose estimation (COCO).
• An efficient UniFormer variant demonstrated 2-4x higher throughput than lightweight models.

Conclusions:

• UniFormer effectively learns discriminative representations by unifying local and global feature extraction.
• The proposed model offers a powerful and versatile backbone for diverse computer vision applications.
• UniFormer provides a promising direction for developing efficient and high-performing visual models.