Efficient Visual Transformer by Learnable Token Merging.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Transformers and self-attention are prevalent in deep learning, leading to visual transformers for computer vision.
• Existing visual transformers can be computationally intensive, motivating research into more efficient architectures.

Purpose of the Study:

• To propose a novel and compact transformer block, the Transformer with Learnable Token Merging (LTM).
• To reduce computational cost (FLOPs) and inference time of visual transformers while maintaining or improving accuracy.

Main Methods:

• Developed the LTM-Transformer block with a learnable token merging scheme.
• Derived a novel variational upper bound for Information Bottleneck (IB) loss.
• Designed a mask module within LTM blocks to minimize the derived IB loss upper bound.

Main Results:

• Replaced transformer blocks in MobileViT, EfficientViT, ViT, and Swin with LTM-Transformer blocks.
• Achieved significant reductions in FLOPs and inference time across various visual transformer backbones.
• Demonstrated comparable or improved prediction accuracy on computer vision tasks.

Conclusions:

• LTM-Transformer enables the creation of compact and efficient visual transformers.
• The proposed method offers a viable approach to enhance performance and reduce computational demands in deep learning models for computer vision.