ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Mixture-of-Experts (MoE) models offer a divide-and-conquer approach to increase model capacity and scalability.
• Vision Transformers (ViT) are a prominent architecture in computer vision tasks.
• Integrating MoE into ViT (ViMoE) presents potential but faces challenges in optimal configuration and stable performance.

Purpose of the Study:

• To investigate the integration of MoE into ViT (ViMoE) for image classification and semantic segmentation.
• To address the performance sensitivity and unreliable routing issues observed in initial ViMoE configurations.
• To propose methods for stabilizing ViMoE and guiding efficient design through expert routing analysis.

Main Methods:

• Implemented a ViMoE by incorporating MoE layers into the ViT architecture.
• Introduced a shared expert within the MoE structure to learn common knowledge and enhance stability.
• Developed techniques to analyze expert routing behavior to identify specialized and redundant layers.

Main Results:

• Demonstrated that ViMoE can be effectively applied to image classification and semantic segmentation tasks.
• Showcased the stabilizing effect of the shared expert on ViMoE performance.
• Provided insights into expert specialization and routing, enabling identification of critical layers.

Conclusions:

• The proposed stable ViMoE design offers a promising direction for enhancing vision models.
• Analyzing expert routing is crucial for optimizing ViMoE efficiency without compromising accuracy.
• This research provides empirical guidance for future development of vision MoE models.