Training high-performance and large-scale deep neural networks with full 8-bit integers

Area of Science:

• Artificial Intelligence
• Computer Science
• Machine Learning

Background:

• Deep neural network (DNN) quantization compresses models by converting floating-point (FP) to integer (INT) data, aiding memory saving and compute acceleration.
• Current research extends DNN quantization from inference to training, enabling online learning on accelerators.
• Existing methods often leave batch normalization (BN) untouched, resulting in incomplete quantization with high-precision FP components.

Purpose of the Study:

• To develop a unified, complete quantization framework for large-scale DNNs that operates entirely with low bit-width integers during training.
• To address the accuracy limitations of current quantization techniques in full integer training scenarios.
• To enable efficient online learning and inference on resource-constrained devices.

Main Methods:

• Proposed a unified complete quantization framework, WAGEUBN, by decomposing DNN computations and fusing specialized quantization functions.
• Quantized all data paths including Weights (W), Activation (A), Gradient (G), Error (E), Update (U), and Batch Normalization (BN).
• Extended quantization to the Momentum optimizer for a fully integerized training process.

Main Results:

• WAGEUBN achieved competitive accuracy on the ImageNet dataset using ResNet18/34/50 models.
• Demonstrated the first successful full 8-bit integer quantization for large-scale DNN training and inference.
• Showcased potential for bit-wise operations, leading to faster processing, reduced memory costs, and improved energy efficiency.

Conclusions:

• WAGEUBN offers a complete quantization solution for DNNs, enabling full 8-bit integer training and inference.
• The framework significantly enhances computational efficiency and reduces resource requirements.
• This advancement holds substantial potential for efficient portable devices with online learning capabilities.