MC#:ミックス・オブ・エキスパート 大型モデル用のミックス・コンプレッサー
IEEE transactions on pattern analysis and machine intelligence
|February 16, 2026
まとめ
ミックス・オブ・エキスパート (Mixture-of-Experts, MoE) モデルは,MC#を使用して圧縮され,静的定量化と動的剪定を組み合わせます. これは,大規模な言語およびビジョン言語モデルの効率的な展開のために,サイズと計算を大幅に削減します.
科学分野:
- 人工知能 (AI) とは,人工知能 (AI) のことです.
- 機械学習 (Machine Learning) とは,機械学習 (Machine Learning) について学ぶことです.
- コンピュータビジョン コンピュータビジョン
背景:
- ミックス・オブ・エキスパート (MoE) モデルは,大型言語モデル (LLM) と視覚言語モデル (VLM) の効率的なスケーリングを提供します.
- 稀少なアクティベーションにもかかわらず,MoEモデルは,すべてのエキスパートをプレロードし,入力ごとに複数のアクティベーションを行うため,重要なコンピューティングとメモリオーバーヘッドに直面しています.
- 専門家モジュールは,モデルのサイズと推論コストの主要な貢献者であり,展開を妨げています.
研究 の 目的:
- MoE-LLM/VLMの攻撃的な圧縮のための統一フレームワーク (MC#) を開発する.
- ストレージ,ロード,および実行時のコンピューティングオーバーヘッドを減らすために.
- 最小限の精度低下で極端な圧縮を達成するために.
主な方法:
- MC#は,静的な定量化 (プレロード混合精度定量化 (PMQ)) とダイナミックなエキスパートプロニング (オンライントップ任意のプロニング (OTP)) を組み合わせています.
- PMQは,アダプティブビット配分のための線形プログラミングを使用し,エキスパートの重要性と量子化エラーのバランスをとります.
- OTPは,トークン固有のエキスパートアクティベーションをモデル化するためにGumbel-Softmaxサンプリングを使用し,推論中にダイナミックなサブセット選択を可能にします.
主要な成果:
- MC#はDeepSeek-VL2で6.2×の重量削減を達成し,重量あたり平均2.57ビットとなった.
- 16ビットベースラインと比較して,パフォーマンスの劣化は最小でした (5つのマルチモダルのベンチマークで1.7%).
- OTPはさらに,専門家アクティベーションを20%削減し,パフォーマンスの損失は1%未満でした.
結論:
- MC#は,MoE-LLM/VLMを圧縮するための非常に効果的な方法を提供し,モデルのサイズと推論コストを大幅に削減します.
- PMQとOTPの組み合わせにより,大きなMoEモデルの効率的な展開は,実質的な精度損失なしに可能になります.
- MC#は,高度なAIモデルのリソース制限の導入を必要とする実用的なアプリケーションの大きな可能性を示しています.
関連する概念動画
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
355
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
355
Multicompartment Models: Overview
611
Multicompartment models are mathematical constructs that depict how drugs are distributed and eliminated within the body. They segment the body into several compartments, symbolizing various physiological or anatomical areas connected through drug transfer processes such as absorption, metabolism, distribution, and elimination.
These models offer a more comprehensive representation of drug behavior in the body than one-compartment models. They accommodate the complexity of drug distribution,...
These models offer a more comprehensive representation of drug behavior in the body than one-compartment models. They accommodate the complexity of drug distribution,...
611
One-Compartment Open Model: Wagner-Nelson and Loo Riegelman Method for ka Estimation
1.2K
This lesson introduces two critical methods in pharmacokinetics, the Wagner-Nelson and Loo-Riegelman methods, used for estimating the absorption rate constant (ka) for drugs administered via non-intravenous routes. The Wagner-Nelson method relates ka to the plasma concentration derived from the slope of a semilog percent unabsorbed time plot. However, it is limited to drugs with one-compartment kinetics and can be impacted by factors like gastrointestinal motility or enzymatic degradation.
On...
On...
1.2K
Mechanistic Models: Overview of Compartment Models
412
Mechanistic models, a category encompassing both physiological and compartmental modeling, differ from empirical models' approaches to incorporating known factors about the systems being modeled. Empirical models describe data with minimal assumptions, while mechanistic models aim to provide a robust description of available data by specifying assumptions and integrating known factors about the system. Compartmental analysis is a key example of a mechanistic model in pharmacokinetics and...
412
Compartment Models: Single-Compartment Model
3.3K
The single-compartment model serves as a simplified representation of the human body. This model assumes that the body functions as a single, well-mixed open compartment. When a drug is administered intravenously, it enters the body and quickly distributes uniformly. The drug then undergoes biotransformation and elimination, ultimately leaving the body. The volume of this compartment is referred to as the apparent volume of distribution into which the drug can uniformly distribute. In this...
3.3K
Mechanistic Models: Compartment Models in Individual and Population Analysis
288
Mechanistic models are utilized in individual analysis using single-source data, but imperfections arise due to data collection errors, preventing perfect prediction of observed data. The mathematical equation involves known values (Xi), observed concentrations (Ci), measurement errors (εi), model parameters (ϕj), and the related function (ƒi) for i number of values. Different least-squares metrics quantify differences between predicted and observed values. The ordinary least...
288


