Rethinking the Importance of Quantization Bias, Toward Full Low-Bit Training

Area of Science:

• Artificial Intelligence
• Computer Science
• Machine Learning

Background:

• Deep neural networks (DNNs) require significant computational and storage resources.
• Quantization techniques offer a solution to reduce these costs.
• Low-bit precision training (≤ 8 bits) faces challenges due to gradient quantization difficulties.

Purpose of the Study:

• To identify the primary cause of accuracy loss during low-bit DNN training.
• To address the limitations of existing gradient quantization methods, such as stochastic rounding.
• To propose a novel quantization approach for efficient and accurate low-bit training.

Main Methods:

• Investigated the impact of bias and variance in gradient quantization noise.
• Identified asymmetric gradient distribution as a source of quantization bias.
• Developed an adaptive piecewise quantization method and a new Piecewise Fixed Point (PWF) data format.

Main Results:

• Demonstrated that bias, not variance, is the critical factor in training accuracy loss.
• Showcased that stochastic rounding is insufficient to mitigate bias-induced training crashes.
• Achieved 1.9–3.5× speedup in image classification, machine translation, OCR, and text classification with <0.5% accuracy loss.

Conclusions:

• The proposed adaptive piecewise quantization effectively controls gradient quantization bias.
• This method enables 8-bit quantization for all layers in large-scale CNN and RNN training with negligible accuracy loss.
• Represents a significant advancement in efficient deep learning model training.