EmbedFormer: Embedded Depth-Wise Convolution Layer for Token Mixing.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Visual Transformers (ViTs) excel due to their ability to capture spatial and channel information.
• MetaFormer architecture highlights the importance of token and channel mixers in transformer performance.
• Depth-wise Convolution (DwConv) is a viable alternative to self-attention for local feature extraction in transformers.

Purpose of the Study:

• To design a pure convolutional transformer model.
• To investigate and address the operational differences between self-attention and DwConv.
• To introduce EmbedFormer, a novel MetaFormer instance, for improved performance in computer vision tasks.

Main Methods:

• A pure convolutional transformer, EmbedFormer, is designed using DwConv with an embedding layer as the token mixer.
• SEBlock is incorporated into the channel mixer to enhance feature representation.
• The MetaFormer block is instantiated with DwConv and an embedding layer.

Main Results:

• EmbedFormer achieves 81.7% top-1 accuracy on ImageNet-1K classification, outperforming Swin Transformer.
• The model demonstrates superior performance on downstream tasks compared to PoolFormer, ResNet, and DeiT.
• On the COCO dataset, EmbedFormer shows +3.0% box AP and +2.3% mask AP improvements over PoolFormer-S24.
• On the ADE20K dataset, EmbedFormer achieves +1.3% mIoU improvement compared to PoolFormer-S24.

Conclusions:

• EmbedFormer, a pure convolutional transformer, offers a compelling alternative to traditional ViTs.
• The proposed architecture effectively leverages DwConv and embedding layers for enhanced performance.
• EmbedFormer demonstrates strong generalization capabilities across various computer vision benchmarks.