Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep model training on extensive datasets is costly.
• Deep model fusion offers a solution by leveraging pre-existing models.
• Parameter interference and lack of interpretability are key challenges in model fusion.

Purpose of the Study:

• To address parameter interference in deep model fusion.
• To introduce an efficient method for upscaling source models into a Mixture of Experts (MoE) model.
• To improve model performance and accelerate new model development without additional training data.

Main Methods:

• Examined fine-tuning of linear layers using subspace analysis.
• Defined parameter interference as an optimization problem.
• Introduced zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction for model upscaling.

Main Results:

• SMILE upscales models into an MoE architecture without extra data or training.
• Dimensionality expansion effectively manages parameter interference.
• Achieved 98%-99% performance of eight individual ViT models with ~50% extra parameters for full fine-tuning.
• Maintained 99% performance for LoRA fine-tuned Flan-T5 models with only 2% extra parameters.

Conclusions:

• SMILE demonstrates adaptability and scalability across image classification, text generation, and large language models (LLMs).
• The method efficiently leverages knowledge from pre-trained models, overcoming challenges of parameter interference.
• SMILE offers a cost-effective approach to deep model development and fusion.