Attention-Based Batch Normalization for Binary Neural Networks.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Binary Neural Networks (BNNs) utilize discrete {-1,1} activations, differing significantly from full-precision networks.
• Batch Normalization (BN) is vital for BNN performance, yet its role requires specific understanding due to activation constraints.
• Existing research highlights the need for novel BN approaches tailored for BNNs.

Purpose of the Study:

• To introduce a novel Attention-Based Batch Normalization (ABN) method for Binary Neural Networks (BNNs).
• To investigate the impact of self-attention mechanisms within BN layers for BNNs.
• To analyze the performance improvements and stability gains offered by ABN.

Main Methods:

• Developed Attention-Based Batch Normalization (ABN) by incorporating self-attention into BN layers.
• Conducted ablation studies to analyze parameter trade-offs within the ABN method.
• Performed experimental analysis on the effects of ABN on BNN characteristics and performance.

Main Results:

• ABN effectively captures image features and provides activation-like functions, enhancing BNN performance.
• The method increases activation distribution imbalance, contributing to improved accuracy.
• Image classification experiments on CIFAR10, CIFAR100, and TinyImageNet showed ABN consistently outperformed standard BN.

Conclusions:

• Attention-Based Batch Normalization (ABN) offers a significant improvement over standard Batch Normalization for BNNs.
• ABN enhances both the accuracy and stability of BNNs in image classification tasks.
• The proposed method demonstrates robustness across different datasets and network architectures.