Spatial Frequency Modulation for Semantic Segmentation.

Area of Science:

• Computer Vision
• Deep Learning
• Signal Processing

Background:

• High spatial frequency information, including fine details and textures, is vital for accurate semantic segmentation.
• Downsampling layers in neural networks, like strided convolution, can cause aliasing and distortion of high-frequency components, violating the Nyquist-Shannon Sampling Theorem.

Purpose of the Study:

• To propose a novel Spatial Frequency Modulation (SFM) technique to preserve high-frequency details during downsampling and upsampling in deep learning models.
• To introduce Adaptive Resampling (ARS) for modulating high-frequency features to lower frequencies and Multi-Scale Adaptive Upsampling (MSAU) for demodulation and recovery of these features.

Main Methods:

• Spatial Frequency Modulation (SFM) involves modulating high-frequency features to a lower frequency before downsampling using Adaptive Resampling (ARS).
• ARS employs dense sampling to scale high-frequency signals, effectively lowering their frequency.
• Multi-Scale Adaptive Upsampling (MSAU) recovers high-frequency information through non-uniform upsampling, leveraging multi-scale information interactions.

Main Results:

• Feature visualization confirms SFM effectively alleviates aliasing and retains fine details after demodulation.
• SFM significantly enhances state-of-the-art segmentation models, achieving +1.5 mIoU on Mask2Former-Swin-T and +1.4 mIoU on InternImage-T on the ADE20K dataset.
• ARS improves Deformable Convolution performance by +0.8 mIoU on Cityscapes, maintaining positional order during non-uniform sampling.

Conclusions:

• The proposed SFM, ARS, and MSAU modules seamlessly integrate with various architectures (CNNs, Transformers) and effectively preserve high-frequency information for improved semantic segmentation.
• SFM demonstrates broad applicability, enhancing performance in image classification, adversarial robustness, instance segmentation, and panoptic segmentation tasks.
• The method successfully addresses the aliasing problem inherent in downsampling operations, leading to substantial performance gains in computer vision tasks.