PSLT: A Light-Weight Vision Transformer With Ladder Self-Attention and Progressive Shift

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Vision Transformers (ViT) excel at modeling long-range dependencies in visual tasks.
• Standard ViTs demand substantial computational resources due to global self-attention mechanisms.
• There is a need for efficient transformer architectures in computer vision.

Purpose of the Study:

• To develop a lightweight transformer backbone with reduced computational requirements.
• To introduce a novel ladder self-attention block and progressive shift mechanism for efficiency.
• To enable efficient modeling of long-range interactions in visual data.

Main Methods:

• Proposed a ladder self-attention block with multiple branches for local self-attention modeling.
• Introduced a progressive shift mechanism to expand the receptive field and enable inter-branch interaction.
• Split input features across branches and used pixel-adaptive fusion to reduce computational cost by approximately 75%.

Main Results:

• The Progressive Shift Ladder Transformer (PSLT) demonstrates effectiveness across image classification, object detection, and person re-identification.
• PSLT achieved 79.9% top-1 accuracy on ImageNet-1k using only 9.2M parameters and 1.9G FLOPs.
• PSLT's performance is comparable to models with over 20M parameters and 4G FLOPs, highlighting its efficiency.

Conclusions:

• The proposed ladder self-attention block and progressive shift mechanism effectively reduce computational costs while maintaining performance.
• PSLT offers a computationally efficient alternative to standard Vision Transformers for various vision tasks.
• The architecture provides a favorable trade-off between performance and resource requirements.