An empirical study of LLaMA3 quantization: from LLMs to MLLMs.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision
• Natural Language Processing

Background:

• LLaMA models are powerful open-source large language models (LLMs) and multi-modal large language models (MLLMs).
• LLaMA3 models demonstrate superior performance due to extensive pre-training.
• Low-bit quantization is crucial for deploying LLMs in resource-constrained environments.

Purpose of the Study:

• To evaluate the performance of LLaMA3 models under low-bit quantization.
• To identify challenges and insights for quantizing LLaMA3 and future LLMs.
• To assess the impact of quantization on both LLMs and MLLMs.

Main Methods:

• Comprehensive evaluation of 10 post-training quantization and LoRA fine-tuning (LoRA-FT) methods on LLaMA3 across 1-8 bits.
• Assessment of LLaMA3-based LLaVA-Next-8B model performance using post-training quantization at 2-4 ultra-low bits.
• Utilized various datasets to reveal low-bit quantization performance characteristics.

Main Results:

• LLaMA3 exhibits non-negligible performance degradation in linguistic and visual tasks when quantized to low bit-widths.
• Performance degradation is particularly pronounced at ultra-low bit widths (2-4 bits).
• A significant performance gap exists at low bit-widths, indicating challenges in LLM compression.

Conclusions:

• LLaMA3's effectiveness is notably reduced by low-bit quantization, especially at extreme low bit-widths.
• Current quantization methods struggle to maintain performance for LLaMA3 in both language and vision tasks.
• Future research must focus on bridging the performance gap to enhance the practicality of quantized LLMs and MLLMs.