Toward Accurate Binarized Neural Networks With Sparsity for Mobile Application

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Binarized Neural Networks (BNNs) offer computational efficiency but often use suboptimal symmetric sign functions for activation binarization.
• Existing BNNs typically binarize activations to -1 and +1 using a fixed threshold of 0, limiting their performance.
• The optimality of fixed thresholds in BNNs has been largely overlooked, creating a performance gap with full-precision networks.

Purpose of the Study:

• To introduce a novel binarization strategy that better approximates the ReLU activation function using 1-bit.
• To develop a BNN variant that enhances gradient propagation through trainable thresholds.
• To establish a new state-of-the-art in BNN performance across mainstream architectures.

Main Methods:

• Proposed Sparsity-inducing BNN (Si-BNN) for quantizing activations to 0 or +1.
• Introduced trainable thresholds within the backward pass of the binarization process.
• Evaluated Si-BNN on AlexNet, ResNet-18, and ResNet-50 architectures.

Main Results:

• Si-BNN achieved state-of-the-art results on binarized networks: Top-1 accuracy of 50.5% for AlexNet, 62.2% for ResNet-18, and 68.3% for ResNet-50.
• Demonstrated a significant reduction in the performance gap between full-precision and BNNs.
• Si-BNN maintains inference efficiency through bit-wise operations, with CPU performance competitive with GPU frameworks.

Conclusions:

• The proposed 0/+1 quantization strategy with trainable thresholds offers a superior approach to BNN activation binarization.
• Si-BNN significantly advances the performance of BNNs, making them more competitive with traditional deep learning models.
• The method provides substantial efficiency gains without compromising accuracy, enabling high-performance deep learning on resource-constrained devices.