BitNet v2: Low-bit and Sparsified Activations for 1-bit Large Language Models

Area of Science:

• Artificial Intelligence
• Computer Science
• Machine Learning

Background:

• Efficient deployment of 1-bit Large Language Models (LLMs) is crucial for broader accessibility.
• Activation outliers present significant challenges for low-bit quantization, hindering LLM performance.
• Existing methods struggle to effectively quantize LLM activations without performance degradation.

Purpose of the Study:

• To investigate sparse and low-bit activation strategies for 1-bit LLMs.
• To develop novel architectures mitigating quantization errors caused by activation outliers.
• To enable fully 4-bit activations for 1-bit LLMs, leveraging next-generation GPU capabilities.

Main Methods:

• Introduced BitNet v2 a4.8, a hybrid quantization-sparsification architecture.
• Utilized 4-bit quantization for specific layer inputs and 8-bit for sparsified intermediate activations.
• Proposed the $\mathcal{H}$ module for online Hadamard transformation to regularize activation distributions.
• Developed BitNet v2 framework for fully 4-bit activations in 1-bit LLMs.

Main Results:

• BitNet v2 a4.8 demonstrated performance comparable to BitNet b1.58 with similar training costs.
• BitNet v2 trained with 8-bit activations matched BitNet b1.58 performance.
• Training BitNet v2 directly with native 4-bit activations resulted in marginal performance loss.
• Significant reduction in computational overhead during batched inference was observed with 4-bit activations.

Conclusions:

• The proposed BitNet v2 architectures effectively address activation outliers in 1-bit LLMs.
• 4-bit activations are feasible for 1-bit LLMs, offering substantial computational benefits.
• These advancements pave the way for more efficient and accessible LLM deployment.