Spectral-Adaptive Modulation Networks for Visual Perception.

Area of Science:

• Computer Vision
• Machine Learning
• Signal Processing

Background:

• 2D convolution and self-attention have distinct spectral behaviors impacting vision model performance.
• Limited theoretical understanding exists for convolution's high-pass filtering advantage and kernel size's effect on shape bias compared to self-attention.

Purpose of the Study:

• To theoretically simulate and compare the frequency responses of 2D convolution and self-attention within a unified framework.
• To identify key factors influencing spectral properties and leverage them for improved vision model design.

Main Methods:

• Employed graph spectral analysis to compare frequency responses of 2D convolution and self-attention.
• Introduced a spectral-adaptive modulation (SPAM) mixer with multi-scale convolutional kernels and spectral re-scaling.
• Developed SPANetV2, a novel vision backbone incorporating the SPAM mixer.

Main Results:

• Node connectivity, modulated by window size, significantly shapes spectral functions, unifying previous findings.
• SPANetV2 demonstrated superior performance compared to state-of-the-art models.
• SPANetV2 achieved state-of-the-art results on ImageNet-1K classification, COCO object detection, and ADE20K semantic segmentation.

Conclusions:

• Graph spectral analysis provides a unified framework for understanding convolution and self-attention spectral properties.
• The spectral-adaptive modulation (SPAM) mixer effectively refines spectral components in visual features.
• SPANetV2 represents a significant advancement in vision backbones, offering improved performance across diverse computer vision tasks.