GSB: Group superposition binarization for vision transformer with limited training samples.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Vision Transformers (ViT) excel in computer vision but suffer from large parameter counts, leading to overfitting with limited training data and high computational demands.
• Model binarization, a compression technique, reduces model size and complexity by using 1-bit parameters and activations, offering a potential solution for ViT limitations.

Purpose of the Study:

• To investigate the effectiveness of model binarization for Vision Transformers (ViT).
• To address the challenges of applying existing binarization techniques to ViTs, particularly information loss in attention modules and value vectors.
• To propose a novel binarization technique, Group Superposition Binarization (GSB), to improve ViT performance.

Main Methods:

• Developed Group Superposition Binarization (GSB), a novel technique tailored for Vision Transformers.
• Investigated and derived improved gradient calculation equations for the binarization process to mitigate gradient mismatch.
• Incorporated knowledge distillation to further enhance the performance of the binarized ViT models.

Main Results:

• Existing CNN binarization methods do not transfer well to ViTs, with accuracy decline attributed to information loss in attention and value vectors.
• The proposed GSB technique effectively addresses these issues, improving the accuracy of binarized ViTs.
• Experiments on limited-data datasets show GSB achieving state-of-the-art performance among binarization schemes, outperforming full-precision ViTs on certain metrics.

Conclusions:

• Model binarization, particularly GSB, is a viable strategy for compressing Vision Transformers, mitigating overfitting, and reducing computational load.
• GSB demonstrates superior performance compared to existing binarization methods and even full-precision ViTs in resource-constrained scenarios.
• The binarization process inherently provides regularization, aiding in training with insufficient data.