Adaptive Locality Guidance: Using Locality Guidance to Initialize the Learning of Vision Transformers on Tiny...

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Vision transformers (VTs) struggle with small datasets, often learning only global information.
• Convolutional neural networks (CNNs) are preferred when extensive training data is unavailable.
• Locality guidance (LG) uses a pre-trained CNN to guide VTs in learning local features.

Purpose of the Study:

• To address the limitation of LG hindering VTs' global feature learning.
• To propose an improved method, adaptive LG (ALG), for training VTs on small datasets.
• To enhance VT performance and reduce computational overhead during training.

Main Methods:

• Developed adaptive LG (ALG), an enhancement over LG.
• ALG uses LG for initialization, then allows VTs to learn independently.
• Feature distance between VT and guidance CNN determines LG duration.
• ALG is a plug-and-play method applicable to various VTs and datasets.

Main Results:

• ALG significantly reduces the computational cost of LG (37%-64%).
• ALG increases validation accuracy of VTs by up to 6.71%.
• The method was successfully applied across ten VTs and five datasets.

Conclusions:

• Adaptive LG effectively balances local and global feature learning in VTs.
• ALG offers a more efficient and effective approach for training VTs on limited data.
• The proposed method enhances VT performance without substantial computational penalties.