ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Large Language Models (LLMs) require substantial computational resources for finetuning.
• Existing finetuning methods often necessitate high-end hardware, limiting accessibility.
• Low-precision quantization offers a path to reduce memory footprints but presents finetuning challenges.

Purpose of the Study:

• To develop a memory-efficient finetuning algorithm for LLMs.
• To enable finetuning of LLMs with 65B parameters on consumer-grade GPUs.
• To integrate arbitrary weight quantizers with low-rank adaptation for flexible finetuning.

Main Methods:

• Proposed ModuLoRA (modular low-rank adaptation), a novel finetuning approach.
• Implemented a quantization-agnostic backward pass for adaptive low-precision weight materialization.
• Integrated state-of-the-art 2-bit QuIP# and 3-bit OPTQ quantization methods.

Main Results:

• Successfully finetuned LLMs with 65B parameters using 2/3/4-bit precision on a single 24GB GPU.
• Achieved competitive performance on text classification, natural language inference, and instruction following tasks.
• Surpassed state-of-the-art ROUGE scores in summarization tasks, outperforming existing 4-bit and 8-bit methods.

Conclusions:

• ModuLoRA significantly reduces memory requirements for LLM finetuning.
• The method enables finetuning of highly precise LLMs (2-bit, 3-bit) for the first time.
• Released ModuLoRA and low-precision models via the LLMTools library for broader accessibility.