CrossFormer++: A Versatile Vision Transformer Hinging on Cross-Scale Attention.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Existing vision transformers do not explicitly leverage multi-scale features crucial for visual perception.
• Handling features at different scales remains a challenge in current transformer architectures.

Purpose of the Study:

• To propose a novel vision transformer architecture, CrossFormer, that effectively integrates multi-scale features.
• To address performance limitations in vision transformers related to self-attention mechanisms and feature representation.

Main Methods:

• Introduced a cross-scale embedding layer (CEL) to blend tokens with multi-scale patches.
• Developed a long-short distance attention (LSDA) mechanism to manage computational load and retain scale-specific features.
• Proposed a progressive group size (PGS) paradigm and an amplitude cooling layer (ACL) to mitigate enlarging self-attention maps and amplitude explosion.

Main Results:

• CrossFormer effectively incorporates cross-scale features, enhancing self-attention capabilities.
• LSDA reduces computational complexity while preserving both small- and large-scale features.
• PGS and ACL successfully address issues of enlarging self-attention maps and amplitude explosion, respectively.

Conclusions:

• CrossFormer++ significantly outperforms existing vision transformers on image classification, object detection, instance segmentation, and semantic segmentation.
• The proposed architecture and techniques offer a more robust and efficient approach to vision transformer design for diverse visual tasks.