PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inference

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Vision Transformers (ViTs) face deployment challenges due to large model size and quadratic complexity with token count.
• Existing methods for accelerating ViTs, like pruning and token reduction, often use fixed ratios and neglect joint optimization, leading to accuracy loss.

Purpose of the Study:

• To introduce PRANCE, a novel framework for jointly optimizing activated channels and tokens on a per-sample basis to accelerate ViT inference.
• To address the challenges of dynamic channel computation and the vast decision space in joint optimization.

Main Methods:

• Developed a meta-network with weight-sharing for dynamic channel support in Multi-Head Self-Attention (MHSA) and Multi-Layer Perceptron (MLP) layers.
• Employed Proximal Policy Optimization (PPO) via a lightweight selector to efficiently manage the combinatorial optimization problem.
• Introduced a 'Result-to-Go' training mechanism modeling ViT inference as a Markov decision process to reduce action space and reward delay.

Main Results:

• Achieved approximately 50% reduction in FLOPs (Floating Point Operations).
• Retained only about 10% of input tokens.
• Maintained lossless Top-1 accuracy, demonstrating significant efficiency gains.

Conclusions:

• PRANCE offers a unified approach to accelerate ViTs by simultaneously optimizing architecture and data.
• The framework effectively tackles the trade-off between compression and accuracy, enabling efficient deployment of ViTs.