MC#:专家混合的大型机型的混合压缩机
IEEE transactions on pattern analysis and machine intelligence
|February 16, 2026
概括
混合专家 (MoE) 模型使用MC#进行压缩,结合静态量化和动态修剪. 这大大降低了大小和计算,以有效地部署大型语言和视觉语言模型.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 计算机视觉 计算机视觉
背景情况:
- 专家组合 (MoE) 模型为大型语言模型 (LLM) 和视觉语言模型 (VLM) 提供了高效的扩展.
- 尽管激活很少,但MoE模型面临着大量的计算和内存开销,因为预先加载所有专家并激活每个输入中的多个.
- 专家模块是模型大小和推断成本的主要贡献者,阻碍了部署.
研究的目的:
- 开发一个统一的框架 (MC#) 用于MoE-LLMs/VLMs的积极压缩.
- 为了减少存储,加载和运行时的计算开销.
- 为了实现极端的压缩与最小的精度退化.
主要方法:
- MC#结合了静态量化 (预加载混合精度量化 - PMQ) 和动态专家修剪 (在线顶部任何修剪 - OTP).
- PMQ使用线性编程进行自适应位分配,平衡专家的重要性和量化错误.
- OTP使用Gumbel-Softmax采样来模拟令牌特定的专家激活,从而在推理过程中实现动态子集选择.
主要成果:
- 在DeepSeek-VL2上,MC#实现了6.2倍的重量减轻,平均每重量为2.57位.
- 与16位基线相比,性能降低是最小的 (1.7%在五个多式联络基准中).
- OTP进一步将专家激活率降低了20%,性能损失不到1%.
结论:
- MC#提供了一种非常有效的方法来压缩MoE-LLMs/VLMs,大大减少模型大小和推断成本.
- 结合PMQ和OTP,可以高效地部署大型MoE模型,而不会造成严重的准确性损失.
- 在实际应用中,MC#显示出了巨大的潜力,这些应用需要先进的人工智能模型的资源有限的部署.
相关概念视频
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
355
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
355
Multicompartment Models: Overview
611
Multicompartment models are mathematical constructs that depict how drugs are distributed and eliminated within the body. They segment the body into several compartments, symbolizing various physiological or anatomical areas connected through drug transfer processes such as absorption, metabolism, distribution, and elimination.
These models offer a more comprehensive representation of drug behavior in the body than one-compartment models. They accommodate the complexity of drug distribution,...
These models offer a more comprehensive representation of drug behavior in the body than one-compartment models. They accommodate the complexity of drug distribution,...
611
One-Compartment Open Model: Wagner-Nelson and Loo Riegelman Method for ka Estimation
1.2K
This lesson introduces two critical methods in pharmacokinetics, the Wagner-Nelson and Loo-Riegelman methods, used for estimating the absorption rate constant (ka) for drugs administered via non-intravenous routes. The Wagner-Nelson method relates ka to the plasma concentration derived from the slope of a semilog percent unabsorbed time plot. However, it is limited to drugs with one-compartment kinetics and can be impacted by factors like gastrointestinal motility or enzymatic degradation.
On...
On...
1.2K
Mechanistic Models: Overview of Compartment Models
412
Mechanistic models, a category encompassing both physiological and compartmental modeling, differ from empirical models' approaches to incorporating known factors about the systems being modeled. Empirical models describe data with minimal assumptions, while mechanistic models aim to provide a robust description of available data by specifying assumptions and integrating known factors about the system. Compartmental analysis is a key example of a mechanistic model in pharmacokinetics and...
412
Compartment Models: Single-Compartment Model
3.3K
The single-compartment model serves as a simplified representation of the human body. This model assumes that the body functions as a single, well-mixed open compartment. When a drug is administered intravenously, it enters the body and quickly distributes uniformly. The drug then undergoes biotransformation and elimination, ultimately leaving the body. The volume of this compartment is referred to as the apparent volume of distribution into which the drug can uniformly distribute. In this...
3.3K
Mechanistic Models: Compartment Models in Individual and Population Analysis
288
Mechanistic models are utilized in individual analysis using single-source data, but imperfections arise due to data collection errors, preventing perfect prediction of observed data. The mathematical equation involves known values (Xi), observed concentrations (Ci), measurement errors (εi), model parameters (ϕj), and the related function (ƒi) for i number of values. Different least-squares metrics quantify differences between predicted and observed values. The ordinary least...
288


