Sparsity-control ternary weight networks.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks (DNNs) demand significant memory and computational resources, limiting their use on edge devices.
• Low-bit weight DNNs offer a solution, with ternary weight networks {-1, 0, +1} being particularly efficient by avoiding multiplications.

Purpose of the Study:

• To develop the first method for controlling sparsity in ternary weight DNNs.
• To improve the efficiency and deployment of DNNs on resource-constrained hardware.

Main Methods:

• Propose a sparsity-control approach (SCA) using a weight discretization regularizer (WDR).
• SCA utilizes a controller parameter α to manage ternary weight sparsity without gradient estimators.

Main Results:

• Demonstrate a positive correlation between the controller α and the sparsity of trained ternary weights.
• SCA consistently outperforms existing methods on benchmark datasets.
• Achieve performance comparable to full-precision DNNs.

Conclusions:

• SCA provides a simple, effective, and novel method for training sparse ternary weight DNNs.
• This approach enhances the practicality of DNNs for resource-limited applications.
• The proposed method offers significant improvements over state-of-the-art techniques.