Scalar Quantization as Sparse Least Square Optimization

Quantization aims to form new vectors or matrices with shared values close to the original. In recent years, the popularity of scalar quantization has been soaring as it is found huge utilities in reducing the resource cost of neural networks. Popular clustering-based techniques suffers substantially from the problems of dependency on the seed, empty or out-of-the-range clusters, and high time complexity. To overcome the problems, in this paper, scalar quantization is examined from a new perspective, namely sparse least square optimization. Specifically, several quantization algorithms based on l₁ least square are proposed and implemented. In addition, similar schemes with l₁ + l₂ and l₀ regularization are proposed. Furthermore, to compute quantization results with given amount of values/clusters, this paper proposes an iterative method and a clustering-based method, and both of them are built on sparse least square optimization. The algorithms proposed are tested under three data scenarios and their computational performance, including information loss, time consumption, and distribution of values of sparse vectors are compared. The paper offers a new perspective to probe the area of quantization, and the algorithms proposed are superior especially under bit-width reduction scenarios, where the required post-quantization resolution (the number of values) is not significantly lower than the original scalar.