Latent Weight Quantization for Integerized Training of Deep Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Integerized training accelerates deep learning using low-bitwidth representations.
• Existing methods neglect full-precision latent weights, causing high memory consumption during gradient updates.
• This memory overhead limits the efficiency of integerized training for large models.

Purpose of the Study:

• To propose the first latent weight quantization schema for general integerized training.
• To minimize quantization perturbation and suppress noise in the training process.
• To enable efficient training and fine-tuning of large deep learning models.

Main Methods:

• Introduced a residual quantization technique to decorrelate latent and integerized weights.
• Developed an optimized dual quantizer with a non-uniform codebook to prevent weight freezing.
• Designed a hardware-friendly, three-segment polyline approximation for codebook optimization.
• Employed importance guidance for optimizing codebooks to minimize weight update disturbance.

Main Results:

• Achieved successful integerized training with 4-bit latent weights across various architectures (ResNets, MobileNetV2, Transformers).
• Demonstrated negligible performance loss in image classification and text generation tasks.
• Successfully fine-tuned Large Language Models (LLMs) up to 13 billion parameters on a single GPU.

Conclusions:

• The proposed latent weight quantization schema significantly enhances the efficiency of integerized training.
• It enables substantial memory reduction, facilitating the training and deployment of large-scale deep learning models.
• This method offers a practical solution for resource-constrained deep learning applications.