PRANCE: 適応的なViT推論のための共同トークン最適化と構造的チャネル剪定
IEEE transactions on pattern analysis and machine intelligence
|September 2, 2025
まとめ
PRANCEは,サンプルごとにチャンネルとトークンを共同で最適化することで,ビジョントランスフォーマー (ViT) を加速します. このフレームワークは,精度を犠牲にすることなく,コンピューティングの複雑性とモデルのサイズを削減し,ViTの効率的な展開を可能にします.
科学分野:
- コンピュータ・ビジョン
- 人工知能
- 機械学習
背景:
- ビジョン・トランスフォーマー (ViT) は,大きなモデルサイズとトークン数による二次的な複雑さにより,展開の課題に直面しています.
- 剪定やトークン削減などのViTを加速するための既存の方法は,固定された比率を使用し,関節最適化を無視し,正確性の損失につながります.
研究 の 目的:
- ViT推論を加速するために,サンプルごとに活性化されたチャネルとトークンを共同で最適化するための新しいフレームワークであるPRANCEを導入します.
- ダイナミックチャネルコンピューティングの課題と 共同最適化における広大な意思決定空間に対処するために
主な方法:
- マルチヘッド・セルフ・アテンション (MHSA) とマルチレイヤー・パーセプトン (MLP) のレイヤでのダイナミックチャネルサポートのための重量共有メタネットワークを開発しました.
- 軽量な選択器を介して近接政策最適化 (PPO) を採用し,組み合わせ最適化問題を効率的に管理する.
- 行動空間と報酬の遅延を減らすために,マルコフ決定プロセスとしてViT推論をモデル化した"結果から開始する"トレーニングメカニズムを導入しました.
主要な成果:
- 約50%のフローティングポイントの削減を達成しました.
- 入力トークンの約10%のみを保持しました.
- 損失のないトップ1の精度を維持し,重要な効率の向上を示しています.
結論:
- PRANCEは,アーキテクチャとデータを同時に最適化することで,ViTを加速させるための統一されたアプローチを提供します.
- このフレームワークは,圧縮と精度とのトレードオフを効果的に解決し,ViTの効率的な展開を可能にします.
関連する概念動画
Transformers with Off-Nominal Turns Ratios
205
In scenarios involving parallel transformers with disparate ratings, developing per-unit models requires accommodating off-nominal turns ratios. This situation arises when the selected base voltages are not proportional to the transformer’s voltage ratings. Consider a transformer where the rated voltages are related by the term a. If the chosen voltage bases satisfy a relationship involving term b, term c is defined as the ratio of these bases. This ratio is then substituted into the...
205
Vision
55.3K
Vision is the result of light being detected and transduced into neural signals by the retina of the eye. This information is then further analyzed and interpreted by the brain. First, light enters the front of the eye and is focused by the cornea and lens onto the retina—a thin sheet of neural tissue lining the back of the eye. Because of refraction through the convex lens of the eye, images are projected onto the retina upside-down and reversed.
55.3K
Transformers in Distribution System
156
Transformers in distribution systems can be broadly categorized into distribution substation transformers and other distribution transformers. They are crucial for stepping down high transmission voltages to levels suitable for distribution and end-user applications.
Distribution substation transformers come in various ratings and typically use mineral oil for insulation and cooling. To prevent moisture and air from entering the oil, some transformers use an inert gas like nitrogen to fill the...
Distribution substation transformers come in various ratings and typically use mineral oil for insulation and cooling. To prevent moisture and air from entering the oil, some transformers use an inert gas like nitrogen to fill the...
156
Improving Translational Accuracy
11.8K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
11.8K
Reducing Line Loss
193
In a three-phase circuit, line loss is an indicator of energy dissipated as heat due to the resistance of transmission lines. To address this, incorporating transformers into the system—a step-up transformer at the source and a step-down transformer at the load—is a strategic solution. Two three-phase transformers are introduced to improve this.
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss...
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss...
193
The Ideal Transformer
878
In single-phase two-winding transformers, two windings are coiled around a magnetic core characterized by cross-sectional area A and magnetic permeability μ. A phasor current i1 enters the left winding while i2 exits the right winding, establishing the fundamental working of the transformer through electromagnetic principles.
Ampere's Law forms the basis of understanding the magnetic field within the transformer. It states that the integral of the magnetic field intensity's...
Ampere's Law forms the basis of understanding the magnetic field within the transformer. It states that the integral of the magnetic field intensity's...
878


