Expediting Large-Scale Vision Transformer for Dense Prediction Without Fine-Tuning.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Large-scale Vision Transformers achieve state-of-the-art results in dense prediction but are computationally expensive.
• Existing acceleration methods primarily focus on image classification, not dense prediction tasks.
• There is a need for efficient Vision Transformer architectures for dense prediction without requiring task-specific fine-tuning.

Purpose of the Study:

• To develop a novel, fine-tuning-free approach for accelerating Vision Transformers in dense prediction tasks.
• To introduce non-parametric operators that reduce computational cost while maintaining performance.
• To demonstrate the versatility of the proposed method across a wide range of dense prediction applications.

Main Methods:

• A token clustering layer is proposed to reduce the number of tokens by grouping neighboring representations, creating low-resolution feature maps.
• Transformer layers are applied exclusively to these condensed, low-resolution tokens to reduce computation.
• A token reconstruction layer is introduced to recover high-resolution representations from the refined low-resolution features.

Main Results:

• The proposed method achieves promising and consistent results across six diverse dense prediction tasks.
• Effectiveness is validated on state-of-the-art open-vocabulary recognition methods.
• The approach demonstrates significant computational savings compared to existing representative methods on dense prediction benchmarks.

Conclusions:

• The developed token clustering and reconstruction layers offer an efficient way to accelerate Vision Transformers for dense prediction.
• The fine-tuning-free nature of the approach broadens its applicability across various computer vision tasks.
• This work presents a significant step towards more computationally feasible and high-performing Vision Transformer models for dense prediction.