Factorization Vision Transformer: Modeling Long-Range Dependency With Local Window Cost.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Transformers offer powerful image representation but face computational challenges, scaling quadratically with resolution.
• The Swin Transformer uses local windows to reduce computation but limits global dependency modeling and robustness.

Purpose of the Study:

• To develop a vision transformer that balances computational cost with enhanced performance and robustness.
• To overcome the limitations of local window-based self-attention in modeling global dependencies and ensuring model robustness.

Main Methods:

• Proposed a novel Factorization Self-Attention (FaSA) mechanism by factorizing the attention matrix into sparse sub-attention matrices.
• Developed the Factorization Vision Transformer (FaViT), a hierarchical model leveraging the FaSA mechanism.
• Achieved linear computational complexity with respect to image resolution.

Main Results:

• FaViT demonstrates high performance in image classification and downstream tasks.
• The model exhibits significant robustness against corrupted and biased data.
• FaViT-B2 improved classification accuracy by 1% and robustness by 7% over Swin-T, with 14% fewer parameters.

Conclusions:

• FaSA provides an effective trade-off between computational cost and performance, enabling long-range dependency modeling.
• FaViT offers a robust and efficient alternative for practical computer vision applications.
• The proposed method shows promise for advancing transformer architectures in vision tasks.