A novel approach to attention mechanism using kernel functions: Kerformer.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Traditional attention mechanisms in AI models have quadratic costs (O(N^2)) concerning sequence length (N).
• This complexity limits efficiency and scalability in tasks involving extended sequences.
• Kerformer is a linear transformer model utilizing a kernel approach.

Purpose of the Study:

• To introduce Kerformer, an AI model designed to overcome the computational limitations of traditional attention mechanisms.
• To enhance efficiency and accuracy in Natural Language Processing (NLP) and vision tasks, particularly those with long sequences.

Main Methods:

• Kerformer employs a nonlinear reweighting mechanism, converting maximum attention into feature-based dot product attention.
• It leverages non-negativity and nonlinear weighting traits of softmax computation for Query (Q) and Key (K) computations.
• The model incorporates an SE Block to further boost performance.

Main Results:

• Attention matrix time complexity is reduced from O(N^2) to O(N).
• Kerformer demonstrates superior time and memory efficiency compared to conventional methods.
• Achieved higher average accuracy (83.39%) in NLP and vision tasks.
• Attained 58.94% average accuracy in long-sequence tasks with improved efficiency and convergence speed in visual tasks.

Conclusions:

• Kerformer offers a scalable and efficient solution for handling long sequences in AI tasks.
• The model significantly reduces computational costs while maintaining or improving accuracy.
• It presents a promising advancement for NLP and computer vision applications facing sequence length limitations.