MetaFormer Baselines for Vision.

Area of Science:

• Computer Vision
• Deep Learning Architectures
• Neural Network Design

Background:

• The MetaFormer architecture, an abstraction of the Transformer, has shown promise in achieving competitive performance.
• Previous research focused on token mixer designs within these architectures.
• This study shifts focus to the inherent capacity of the MetaFormer framework itself.

Purpose of the Study:

• To explore the performance potential of the MetaFormer architecture independent of complex token mixer designs.
• To validate MetaFormer's effectiveness with basic and conventional token mixers.
• To introduce and evaluate a novel activation function, StarReLU.

Main Methods:

• Developed baseline MetaFormer models using simple token mixers: identity mapping (IdentityFormer) and random matrices (RandFormer).
• Implemented ConvFormer using depthwise separable convolutions, comparing it to ConvNeXt.
• Created CAFormer by combining depthwise separable convolutions and self-attention.
• Introduced and tested StarReLU, a variant of Squared ReLU, as an activation function.

Main Results:

• IdentityFormer achieved over 80% accuracy on ImageNet-1K, establishing a solid performance baseline.
• RandFormer surpassed IdentityFormer with over 81% accuracy, showcasing MetaFormer's compatibility with arbitrary mixers.
• ConvFormer outperformed ConvNeXt, and CAFormer achieved a new record of 85.5% accuracy on ImageNet-1K.
• StarReLU reduced activation FLOPs by 71% compared to GELU while improving performance.

Conclusions:

• MetaFormer provides a robust framework that ensures a high performance lower bound, even with minimal token mixers.
• The architecture demonstrates versatility, performing well with diverse and even random token mixers.
• MetaFormer, when combined with conventional or novel components like StarReLU, can achieve state-of-the-art results efficiently.