Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Patchification is the standard image tokenization method for Vision Transformers (ViT).
• This compression reduces sequence length and computational cost but may cause information loss.
• The impact of patch size on visual understanding remains underexplored.

Purpose of the Study:

• To investigate information loss from patchification in visual understanding.
• To explore the relationship between patch size and model performance.
• To establish theoretical foundations for non-compressive vision models.

Main Methods:

• Conducted extensive patch size scaling experiments.
• Evaluated performance across diverse vision tasks, input scales, and architectures (ViT, Mamba).
• Analyzed the impact of patch size on decoder head importance for dense prediction.

Main Results:

• A scaling law was observed: decreasing patch sizes consistently improved predictive performance.
• Optimal performance was achieved with pixel tokenization (1x1 patches).
• Smaller patches reduced the necessity of task-specific decoder heads for dense prediction.
• Achieved 84.6% accuracy on ImageNet-1k with a sequence length of 50,176 tokens.

Conclusions:

• Pixel tokenization (1x1 patches) is beneficial for visual understanding across various models and tasks.
• This finding challenges the necessity of compressive encoding in vision models.
• The study provides insights for developing future non-compressive vision architectures.