ToFe: Lagged Token Freezing and Reusing for Efficient Vision Transformer Inference.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Vision Transformers (ViTs) excel in vision tasks but are computationally intensive.
• Resource-constrained edge devices limit ViT deployment due to high computational costs.
• Existing token reduction methods irreversibly discard tokens, hindering potential later use.

Purpose of the Study:

• To develop an efficient Vision Transformer framework for resource-constrained edge devices.
• To address the irreversible token discarding issue in current transformer efficiency methods.
• To balance model performance with computational overhead for practical applications.

Main Methods:

• Introduced a novel token freezing and reusing (ToFe) framework.
• Developed a prediction module for identifying important tokens and a recovery module for frozen tokens.
• Employed computation budget-aware end-to-end training to optimize token processing adaptively.

Main Results:

• Achieved a 50% reduction in computational cost for the LV-ViT model.
• Maintained performance with less than a 2% drop in Top-1 accuracy.
• Demonstrated a superior tradeoff between performance and complexity compared to existing methods.

Conclusions:

• The ToFe framework enables efficient Vision Transformer deployment on edge devices.
• Token freezing and reusing allows for adaptive computation, reducing costs without significant accuracy loss.
• ToFe offers a practical solution for balancing performance and computational efficiency in transformer models.