GFNet: Global Filter Networks for Visual Recognition.

Area of Science:

• Computer Vision
• Deep Learning Architectures
• Frequency Domain Analysis

Background:

• Vision Transformer (ViT) and multi-layer perceptron (MLP) models show promise but struggle with high-resolution images due to quadratic complexity.
• Scaling self-attention and MLP models becomes computationally prohibitive as image size increases, limiting their application in demanding vision tasks.

Purpose of the Study:

• Introduce the Global Filter Network (GFNet), an efficient architecture for learning long-range spatial dependencies in the frequency domain.
• Develop GFNet models (isotropic and hierarchical) that offer improved accuracy/complexity trade-offs and scalability compared to existing vision models.

Main Methods:

• Replaced self-attention layers with a sequence of 2D discrete Fourier transform, element-wise multiplication with learnable global filters, and 2D inverse Fourier transform.
• Designed isotropic GFNet models with Transformer-style simplicity and hierarchical GFNet models incorporating CNN-style designs.
• Evaluated model performance on image classification (ImageNet-1k, ImageNet-21k) and dense prediction tasks (ADE20k).

Main Results:

• GFNet achieves log-linear complexity, enabling efficient processing of high-resolution features.
• Isotropic GFNet models demonstrate favorable accuracy/complexity trade-offs against ViT and MLP models.
• Hierarchical GFNet models achieve strong performance, including 85.0% top-1 accuracy on ImageNet-1k and 54.3 mIoU on ADE20k, showcasing scalability and effectiveness.

Conclusions:

• GFNet presents a computationally efficient and scalable alternative to current Transformer and CNN architectures for computer vision.
• The frequency domain approach effectively captures long-range spatial dependencies, enhancing generalization ability and robustness.
• GFNet offers a promising direction for developing high-performance vision models with reduced computational cost.