Graph-based vision transformer with sparsity for training on small datasets from scratch.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Vision Transformers (ViTs) excel in large-scale image classification but underperform compared to Convolutional Neural Networks (CNNs) on smaller datasets due to a lack of inductive bias.
• This performance disparity highlights a need for architectural modifications to enhance ViT efficiency in data-limited scenarios.

Purpose of the Study:

• To introduce a Graph-based Vision Transformer (GvT) designed to improve ViT performance on small datasets by integrating graph-based mechanisms.
• To address the limitations of standard ViTs in capturing local spatial information and mitigate the low-rank bottleneck in attention mechanisms.

Main Methods:

• The proposed GvT employs graph convolutional projection for queries and keys, utilizing spatial adjacency matrices within each block.
• Graph convolution is used to generate values, and talking-heads technology is applied to overcome the low-rank bottleneck in attention heads.
• Graph-pooling is integrated between intermediate blocks to reduce token count and enhance semantic information aggregation.

Main Results:

• GvT demonstrates comparable or superior performance to deep CNNs on small datasets.
• The GvT model surpasses the performance of standard ViTs that are not pre-trained on large datasets.

Conclusions:

• The GvT architecture effectively introduces inductive biases, significantly improving Vision Transformer performance on small datasets.
• GvTs offer a promising alternative to traditional CNNs and standard ViTs in scenarios with limited training data.