SiMaN: Sign-to-Magnitude Network Binarization.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Binary neural networks (BNNs) offer computational and storage efficiency but face challenges in discrete weight optimization and bit entropy maximization.
• Current methods often use the sign function for binarization, which can be suboptimal.
• Handling discrete constraints while maximizing entropy is crucial for effective BNN training.

Purpose of the Study:

• To develop a novel weight binarization method for BNNs that addresses the limitations of existing sign-based approaches.
• To achieve high-quality discrete solutions in a computationally efficient manner.
• To improve the performance of BNNs on standard image classification benchmarks.

Main Methods:

• Formulated an angle alignment objective to constrain weight binarization to {0,+1}.
• Developed a sign-to-magnitude network binarization (SiMaN) approach providing an analytical solution.
• Demonstrated that removing L2 regularization effectively addresses the Laplacian distribution of learned weights, enabling entropy maximization.

Main Results:

• SiMaN analytically encodes high-magnitude weights to +1 and others to 0, achieving efficient and high-quality binarization.
• The method proved superior to sign-based state-of-the-art techniques on CIFAR-10 and ImageNet datasets.
• It was shown that learned weights approximate a Laplacian distribution, which can be optimized by removing L2 regularization.

Conclusions:

• SiMaN offers a computationally efficient and effective alternative to sign-based binarization for BNNs.
• The proposed angle alignment objective and removal of L2 regularization lead to improved BNN performance.
• The study provides a new direction for optimizing discrete constraints in binary neural networks.