Vision transformers need zoomer: Efficient ViT with categorical attention prior-guided zoom adapter

Vision Transformers (ViTs) have made significant strides recently, but vanilla ViT models struggle with complex scenes, particularly multi-label images and occluded objects. Humans can extract specific visual cues from complex images to guide effective classification. Inspired by the observation that class-conditional saliency-typically used as a post-hoc explanatory tool-can be converted into an ante-hoc structural prior through distillation, we propose ZoomViT, a novel approach that introduces a Categorical Attention Prior (CAP) to guide adaptive zoom for efficient vision transformers. ZoomViT is based on two key observations: (1) Humans and advanced models can intelligently ignore class-irrelevant areas and focus on semantically important regions through categorical attention prior. (2) Standard ViTs can achieve superior classification accuracy when guided by adaptive zooming into regions that align with categorical attention prior. Our approach introduces the Zoomer, a lightweight adapter with only 0.8M parameters that generates categorical attention prior-guided score maps for image regions and dynamically adjusts patch sizes accordingly. This component leverages the CAP to increase patch density in class-decisive regions before classification inference, converting a traditionally post-hoc saliency signal into an ante-hoc input prior, while using larger patches for less critical areas. The categorical attention prior-guided adaptation enhances both efficiency and accuracy, especially in complex images. Experiments show ZoomViT, based on the DeiT-S framework, achieves 83.8%(+4.0%) top-1 accuracy on ImageNet-1k, surpassing existing efficient state-of-the-art (SOTA) ViTs in accuracy and efficiency. The code is publicly available at https://github.com/chenziwenhaoshuai/ZoomViT.