Patch-Based Separable Transformer for Visual Recognition

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Transformers offer powerful capabilities for visual recognition but suffer from high computational complexity, limiting their widespread deployment.
• Existing transformer architectures struggle with generalization to diverse downstream tasks like object detection and segmentation, unlike Convolutional Neural Networks (CNNs).

Purpose of the Study:

• To introduce a novel transformer-based architecture that addresses computational complexity and enhances transferability across various visual tasks.
• To retain both local and global information processing within the network for improved feature representation.

Main Methods:

• The proposed architecture factorizes the full spatial self-attention mechanism into pixel-wise local attention and patch-wise global attention.
• This factorization significantly reduces computational cost while preserving multi-granularity information, crucial for generating multi-scale features.
• The Separable Transformer (SeT) model is constructed based on this factorized attention mechanism for visual modeling.

Main Results:

• SeT demonstrates superior performance compared to state-of-the-art transformer-based methods.
• The proposed model also surpasses traditional CNN counterparts on key visual recognition tasks.
• Experimental validation was conducted across image classification, object detection, and instance segmentation tasks.

Conclusions:

• The Separable Transformer (SeT) offers an efficient and transferable solution for visual modeling, overcoming limitations of previous transformer architectures.
• SeT's factorized attention mechanism effectively balances computational cost and information retention, enabling strong performance across diverse downstream tasks.