Lightweight transformer image feature extraction network.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Transformer models are increasingly used for image feature extraction.
• Quadratic complexity of self-attention in Transformers limits high-resolution image processing and increases computational cost.
• Existing methods struggle with the computational demands of large-scale image data.

Purpose of the Study:

• To develop efficient Transformer models for image feature extraction.
• To address the quadratic complexity issue in self-attention mechanisms.
• To enable the processing of high-resolution images with reduced computational expense.

Main Methods:

• Reduced the quadratic complexity of the self-attention mechanism to linear complexity.
• Introduced a parameter-less lightweight pruning method to filter unimportant tokens.
• Combined these approaches to create an efficient attention mechanism.

Main Results:

• The combined methods reduced the original Transformer model's computation by 30%-50%.
• The efficient attention mechanism achieved a 60%-70% reduction in computation.
• Demonstrated improved processing capabilities for high-resolution images.

Conclusions:

• The proposed efficient attention mechanism significantly reduces computational load in Transformer models.
• This approach enhances the feasibility of using Transformers for high-resolution image analysis.
• The methods offer a practical solution for computationally expensive image feature extraction tasks.