Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Multi-task learning (MTL) enhances model generalization by training on diverse tasks.
• Task arithmetic methods merge independently fine-tuned models for MTL.
• Static merging approaches struggle with task diversity, leading to performance degradation.

Purpose of the Study:

• To propose a novel Weight-Ensembling Mixture of Experts (WEMoE) for effective multi-task model merging.
• To address limitations of static merging methods in handling task interference and diversity.
• To introduce an efficient variant (E-WEMoE) for reduced computational overhead and parameter count.

Main Methods:

• Identify critical modules in Transformer models based on parameter variations.
• Statically merge non-critical modules and transform critical modules into a Mixture-of-Experts (MoE) structure.
• Dynamically merge MoE expert modules during inference based on input samples for adaptive merging.

Main Results:

• WEMoE and E-WEMoE outperform state-of-the-art (SOTA) model merging techniques.
• Demonstrated superior MTL performance, generalization, and robustness across various architectures and tasks.
• E-WEMoE significantly reduces trainable parameters, overall parameter count, and computational overhead.

Conclusions:

• WEMoE offers a flexible and adaptive approach to multi-task model merging.
• E-WEMoE provides an efficient and effective solution with reduced resource requirements.
• The proposed methods represent a significant advancement in multi-task learning model merging.