What Makes for Good Tokenizers in Vision Transformer?

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Transformers have become dominant in computer vision, replacing convolutional neural networks.
• Self-attention mechanisms in transformers rely on effective tokenization of input data.
• The principles of optimal tokenization for vision transformers remain poorly understood.

Purpose of the Study:

• To investigate the critical role of tokenization in vision transformer performance.
• To develop improved design strategies for vision tokenizers based on information theory.
• To introduce novel, plug-and-play components for enhancing vision transformer tokenization.

Main Methods:

• Analyzed tokenization from an information trade-off perspective.
• Developed Modulation across Tokens (MoTo) to enhance inter-token modeling via normalization.
• Integrated a regularization technique, TokenProp, into the training process.

Main Results:

• MoTo and TokenProp demonstrated improved performance across various transformer architectures.
• These methods offer enhanced properties with negligible computational cost.
• The study validates the significance of tokenizer design in vision transformers.

Conclusions:

• Tokenization is a crucial, often overlooked, component in vision transformers.
• The proposed MoTo and TokenProp offer effective and efficient improvements for vision transformer tokenizers.
• Further research into tokenizer design is warranted for advancing vision transformer capabilities.