Rethinking Attention Mechanisms in Vision Transformers with Graph Structures.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Standard Vision Transformers (ViT) utilize Multi-Head Attention (MHA), which is parameter-intensive and can compromise image locality.
• There is a need for more efficient and effective ViT architectures that preserve spatial information.

Purpose of the Study:

• To propose a novel Vision Transformer architecture, GHA-ViT, incorporating Graph Head Attention (GHA).
• To enhance the performance of ViTs while reducing computational complexity and parameter count.

Main Methods:

• Replaced the Multi-Head Attention (MHA) mechanism in standard ViTs with a novel Graph Head Attention (GHA).
• Applied graph structures to the attention heads of the transformer to better capture relationships within image patches.
• Evaluated GHA-ViT on various datasets including CIFAR-10/100, MNIST, MNIST-F, and ImageNet-1K.

Main Results:

• GHA-ViT demonstrated superior performance over pure ViT models across multiple datasets.
• Achieved a Top-1 accuracy of 81.7% on ImageNet-1K with the GHA-B model (approx. 29M parameters).
• Significantly reduced parameters (17-fold) and improved performance (0.4%/4.3%) on CIFAR-10/100 compared to existing ViTs.

Conclusions:

• The proposed GHA-ViT effectively maintains both locality and globality of image patches, ensuring attention diversity.
• GHA-ViT presents a promising lightweight alternative to current state-of-the-art ViT models, balancing accuracy, parameter count, and computational operations.