Related Experiment Video
Updated: May 9, 2025
03:31
Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications
Published on: December 15, 2023
428
MSCViT: A small-size ViT architecture with multi-scale self-attention mechanism for tiny datasets
1National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, China; Department of Computer Science, Sichuan University, China.
Summary
This study introduces MSCViT, a small Vision Transformer (ViT) architecture designed for limited data. MSCViT enhances performance on tiny datasets by integrating multi-scale attention and convolutional blocks, outperforming standard ViTs.
Area of Science:
- Computer Vision
- Deep Learning
- Artificial Intelligence
Background:
- Vision Transformer (ViT) models excel at long-range dependencies but require massive datasets.
- ViTs underperform Convolutional Neural Networks (CNNs) on small-scale (tiny) datasets due to data requirements for representational capacity.
Purpose of the Study:
- To develop a parameter-efficient Vision Transformer architecture suitable for tiny datasets.
- To improve ViT performance in low-data regimes without pre-training on large datasets.
Main Methods:
- Introduced a small-size ViT architecture (MSCViT) with multi-scale self-attention and convolution blocks.
- Incorporated wavelet convolution for selective high-frequency component extraction and local feature learning.
- Developed a lightweight multi-head attention module to reduce token count and computational load.
- Replaced standard positional encoding with a local feature extraction module.
Main Results:
- MSCViT demonstrates parameter efficiency and suitability for tiny datasets.
- Achieved 84.68% accuracy on CIFAR-100 using only 14.0M parameters and 2.5 GFLOPs.
- Outperformed original ViT on small-scale datasets without large-scale pre-training.
Conclusions:
- The proposed MSCViT architecture effectively addresses the data limitations of standard ViTs.
- MSCViT offers a viable solution for applying advanced deep learning models in resource-constrained environments.
- This work highlights the potential of hybrid architectures combining attention and convolution for efficient feature learning.
