DctViT: Discrete Cosine Transform meet vision transformers.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Vision Transformers (ViTs) excel at capturing long-range dependencies via self-attention.
• Convolutional Neural Networks (CNNs) are effective at extracting local features.
• Hybrid approaches combining CNNs and ViTs offer potential for balanced performance and computational efficiency.

Purpose of the Study:

• To propose a novel hybrid CNN-Transformer network for enhanced vision tasks.
• To introduce a new feature map resolution reduction technique, DCT-Attention Down-sample (DAD).
• To evaluate the performance of the proposed DctViT model on benchmark datasets.

Main Methods:

• Developed a hybrid network integrating CNNs for local feature extraction and Transformers for long-range dependency modeling.
• Proposed the DCT-Attention Down-sample (DAD) module, utilizing Discrete Cosine Transform and self-attention for feature map reduction.
• Trained and evaluated the DctViT-L model on ImageNet 1K and DctViT-B as a backbone for RetinaNet on COCO val2017.

Main Results:

• DctViT-L achieved 84.8% top-1 accuracy on ImageNet 1K, surpassing existing state-of-the-art models like CMT and Next-ViT.
• The DctViT-B backbone improved RetinaNet's mAP to 46.8% on COCO val2017, outperforming CMT-S and SpectFormer.
• Both configurations demonstrated superior performance with lower computational costs compared to baseline models.

Conclusions:

• The proposed hybrid DctViT architecture effectively leverages the strengths of both CNNs and Transformers.
• DAD module offers an efficient method for feature map resolution reduction in vision models.
• DctViT presents a promising direction for developing high-performance, computationally efficient vision systems.