MC#: Mixture Compressor for Mixture-of-Experts Large Models.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Mixture-of-Experts (MoE) models offer efficient scaling for large language models (LLMs) and vision-language models (VLMs).
• Despite sparse activation, MoE models face significant computational and memory overhead due to preloading all experts and activating multiple per input.
• Expert modules are the primary contributors to model size and inference cost, hindering deployment.

Purpose of the Study:

• To develop a unified framework (MC#) for aggressive compression of MoE-LLMs/VLMs.
• To reduce storage, loading, and runtime computational overheads.
• To achieve extreme compression with minimal accuracy degradation.

Main Methods:

• MC# combines static quantization (Pre-Loading Mixed-Precision Quantization - PMQ) and dynamic expert pruning (Online Top-any Pruning - OTP).
• PMQ uses linear programming for adaptive bit allocation, balancing expert importance and quantization error.
• OTP employs Gumbel-Softmax sampling to model token-specific expert activation, enabling dynamic subset selection during inference.

Main Results:

• MC# achieved a 6.2× weight reduction on DeepSeek-VL2, averaging 2.57 bits per weight.
• Performance degradation was minimal (1.7% across five multimodal benchmarks) compared to the 16-bit baseline.
• OTP further reduced expert activation by 20% with less than 1% performance loss.

Conclusions:

• MC# provides a highly effective method for compressing MoE-LLMs/VLMs, drastically reducing model size and inference costs.
• The combination of PMQ and OTP enables efficient deployment of large MoE models without substantial accuracy loss.
• MC# demonstrates significant potential for practical applications requiring resource-constrained deployment of advanced AI models.