Efficient Vision Transformer via Token Merger.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Vision Transformers (ViTs) process images by dividing them into fixed-size patches, creating numerous tokens.
• This tokenization strategy can lead to semantic and visual redundancy, impacting efficiency.
• Existing methods for reducing token count may compromise important contextual information.

Purpose of the Study:

• To introduce a novel module, Token Merger, for accelerating Vision Transformers.
• To address the issue of redundant tokens in ViTs by merging them into a compact representation.
• To enhance the efficiency of ViTs while preserving semantic meaning and contextual cues.

Main Methods:

• Proposes Token Merger, a module that identifies and merges semantically similar tokens.
• Employs meta tokens to represent key image content cues during the merging process.
• Introduces learnable gates to control adaptive token merge ratios across different ViT layers.
• Designed as a plug-and-play module for easy integration into existing ViT architectures.

Main Results:

• Token Merger effectively merges redundant tokens, creating a compact set that retains clear semantics.
• Achieves significant acceleration in inference speed (e.g., 62%) by reducing token count (e.g., 95%).
• Maintains high accuracy, with only a minor drop (e.g., 0.4% on ImageNet classification).
• Demonstrates superior performance and generalization compared to token pruning and other downsampling methods.

Conclusions:

• Token Merger offers an effective solution for accelerating Vision Transformers by intelligently merging redundant tokens.
• The module preserves crucial contextual information, leading to better performance and generalization.
• Provides a favorable trade-off between computational efficiency and model accuracy for vision tasks.