CycleMLP: A MLP-Like Architecture for Dense Visual Predictions

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Multilayer Perceptrons (MLPs) have limitations in dense visual prediction tasks due to sensitivity to image size and quadratic computational complexity.
• Existing advanced MLP architectures like MLP-Mixer, ResMLP, and gMLP are often infeasible for dense prediction tasks.
• Convolutional Neural Networks (CNNs) and Transformers are dominant but can be computationally intensive.

Purpose of the Study:

• To introduce CycleMLP, a versatile neural backbone network for dense visual prediction tasks.
• To address the limitations of existing MLP architectures regarding image size adaptability and computational efficiency.
• To provide a theoretical analysis comparing CycleMLP with convolution and self-attention mechanisms.

Main Methods:

• Developed CycleMLP, a novel MLP architecture utilizing local windows for efficient computation.
• Achieved linear computational complexity with respect to image size, contrasting with the O(N^2) complexity of traditional MLPs.
• Conducted theoretical analysis to understand the relationship between CycleMLP, convolution, and multi-head self-attention.

Main Results:

• CycleMLP demonstrates adaptability to various spatial image sizes.
• Achieved linear computational complexity, making it suitable for large-scale dense prediction tasks.
• Models built with CycleMLP surpass state-of-the-art MLP and Transformer models in performance, using fewer parameters and FLOPs.
• CycleMLP-Tiny outperformed Swin-Tiny by 1.3% mIoU on the ADE20 K dataset.
• Exhibited excellent zero-shot robustness on the ImageNet-C dataset.

Conclusions:

• CycleMLP offers a versatile and efficient backbone for dense visual prediction tasks like object detection, segmentation, and human pose estimation.
• The architecture overcomes the limitations of previous MLPs, enabling broader applicability in computer vision.
• CycleMLP provides a competitive alternative to Transformer models, achieving superior or comparable results with reduced computational cost.