Binary Transformer Based on the Alignment and Correction of Distribution.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Transformers are powerful AI models with high computational demands, limiting their use in resource-constrained embedded systems.
• Binary quantization offers reduced memory and faster computation but faces challenges like distribution shift in lightweight transformers.

Purpose of the Study:

• To develop an effective binary quantization technique for lightweight transformers suitable for embedded intelligent sensors.
• To address the distribution shift problem inherent in current binarization methods for transformers.

Main Methods:

• Investigated feature distribution alignment operations for binarization.
• Designed median shift and mean restore to align binary and full-precision transformer feature distributions.
• Developed a knowledge distillation architecture with a teacher-student model (full-precision and binary transformers) for distribution correction.

Main Results:

• The proposed binary optimization model effectively tackles the distribution shift problem.
• Experimental results on CIFAR10, CIFAR100, ImageNet-1k, and TinyImageNet demonstrate superior performance.
• The model outperforms existing state-of-the-art binarization methods at equivalent computational complexity.

Conclusions:

• The novel binary quantization approach enhances the feasibility of transformers in embedded intelligent sensors.
• The feature distribution alignment and knowledge distillation methods ensure data completeness and accuracy.
• This work presents a significant advancement in efficient deep learning for edge devices.