Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Large language models (LLMs) demonstrate remarkable capabilities across diverse applications.
• Training LLMs demands substantial computational resources, hindering efficiency and scalability.
• Low-precision training techniques have emerged as a key strategy to overcome these resource limitations.

Purpose of the Study:

• To provide a comprehensive survey of existing low-precision training methods for LLMs.
• To systematically categorize these methods based on their numerical formats for clarity.
• To identify future research directions in efficient LLM training.

Main Methods:

• Categorization of low-precision training methods into three groups: fixed-point/integer-based, floating-point-based, and customized format-based.
• Discussion of quantization-aware training approaches and their relation to low-precision training.
• Analysis of robustness and deployment reliability considerations for low-precision models.

Main Results:

• Low-precision training significantly enhances efficiency and scalability of LLMs.
• Numerical format is a critical factor influencing hardware compatibility and computational performance.
• Existing methods are systematically organized, offering a unified overview of the field.

Conclusions:

• Low-precision training is crucial for advancing LLM efficiency and accessibility.
• Further research into numerical formats and robustness can unlock new potential.
• The survey provides a valuable resource for researchers navigating the landscape of efficient LLM training.