DR-EFT: Exploring and reloading domain-representative experts for the memory-constrained fine-tuning of MoE large...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Mixture of Experts (MoE) Large Language Models (LLMs) show high performance but face memory challenges during fine-tuning due to large parameter counts.
• MoE models often utilize a subset of experts for specific domains, suggesting potential for targeted fine-tuning.

Purpose of the Study:

• To develop an efficient fine-tuning framework for MoE LLMs under memory constraints.
• To address challenges like expert activation drift during the fine-tuning process.

Main Methods:

• Proposed DR-EFT (Domain-Representative Experts for Fine-Tuning) framework utilizing a two-stage learning mechanism.
• Employed quantized model fine-tuning to identify representative experts and mitigate activation drift.
• Introduced a novel MoE training dynamic that includes secondary relevant experts, differing from existing pruning methods.

Main Results:

• DR-EFT reduces memory consumption of MoE LLMs by approximately 50% with only marginal performance degradation.
• The framework demonstrated robustness across various downstream tasks and popular MoE LLMs (Qwen, DeepSeek, Ernie).
• Identified the crucial role of secondary relevant experts in domain-specific fine-tuning.

Conclusions:

• DR-EFT offers an effective solution for memory-efficient fine-tuning of MoE LLMs.
• The inclusion of secondary relevant experts enhances the theoretical understanding of domain-specific expert selection.
• The proposed method provides a practical approach for adapting large MoE models in resource-limited environments.