FsaNet: Frequency Self-Attention for Semantic Segmentation.

Area of Science:

• Computer Vision
• Deep Learning
• Image Processing

Background:

• Self-attention mechanisms are crucial in deep learning for computer vision.
• Traditional self-attention models exhibit high computational complexity, limiting their efficiency.
• Processing images across all frequency bands can be computationally intensive.

Purpose of the Study:

• To develop a computationally efficient self-attention mechanism by leveraging spectral properties.
• To introduce a novel plug-and-play module, FsaNet, for Convolutional Neural Network (CNN) architectures.
• To demonstrate the effectiveness of low-frequency self-attention for image analysis tasks.

Main Methods:

• Proposing a new self-attention mechanism operating on frequency components of images.
• Implementing individualized processing over different frequency bands, focusing on low-frequency components.
• Designing FsaNet as a plug-and-play module for CNNs, simplifying token mapping and mixing.

Main Results:

• FsaNet achieves significant reductions in memory usage (87-90%), FLOPs (96-98%), and runtime (97-98%) compared to regular self-attention.
• Achieved state-of-the-art 83.0% mIoU on the Cityscape dataset with a ResNet101 backbone.
• Demonstrated competitive results on ADE20k and VOCaug datasets, and enhanced Mask R-CNN for instance segmentation.

Conclusions:

• Frequency self-attention, particularly low-frequency processing, offers a highly efficient alternative to traditional self-attention.
• FsaNet modules can be integrated into various CNN architectures (e.g., Segformer) to boost performance.
• The proposed method shows potential for improving performance even without network retraining, highlighting its robustness.