Uni-MoE: Scaling Unified Multimodal LLMs With Mixture of Experts.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Advancements in Multimodal Large Language Models (MLLMs) require scalable solutions, but often face high computational costs.
• Existing Mixture of Experts (MoE) architectures for scaling models have limitations in the number of experts and modalities handled.

Purpose of the Study:

• To develop a pioneering unified Multimodal Large Language Model (MLLM) using the Mixture of Experts (MoE) architecture, named Uni-MoE.
• To enable efficient handling of a wide array of modalities within a single MLLM framework.

Main Methods:

• Developed Uni-MoE featuring modality-specific encoders with connectors for unified multimodal representation.
• Implemented a sparse MoE architecture for efficient training and inference via data and model parallelism.
• Introduced a progressive training strategy including cross-modality alignment, modality-specific expert training, and Low-Rank Adaptation (LoRA) tuning.

Main Results:

• Uni-MoE demonstrated a significant reduction in performance bias when handling mixed multimodal datasets.
• The model exhibited improved multi-expert collaboration and enhanced generalization capabilities.
• Experimental results confirmed the effectiveness of the proposed training strategy.

Conclusions:

• Uni-MoE presents an effective and scalable solution for unified multimodal processing in LLMs.
• The proposed architecture and training methodology address the computational challenges and performance biases in MLLMs.
• This work paves the way for more efficient and versatile multimodal AI systems.