Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Self-attention mechanisms are crucial for deep feature representation in visual tasks, enabling capture of long-range dependencies within samples.
• However, self-attention exhibits quadratic complexity and overlooks inter-sample correlations, posing computational challenges.

Purpose of the Study:

• To propose a novel attention mechanism, termed external attention, that addresses the limitations of self-attention.
• To develop an efficient and effective attention method with linear complexity for various visual tasks.

Main Methods:

• Introduced external attention using two small, learnable, shared external memories.
• Implemented external attention with cascaded linear and normalization layers, enabling easy replacement of self-attention.
• Incorporated multi-head mechanism to create an all-MLP architecture, External Attention MLP (EAMLP), for image classification.

Main Results:

• External attention demonstrates linear complexity, implicitly considering correlations between all data samples.
• The EAMLP architecture achieved results comparable or superior to self-attention variants across diverse tasks.
• The proposed method significantly reduced computational and memory costs compared to self-attention.

Conclusions:

• External attention offers a computationally efficient alternative to self-attention for deep feature representation in visual tasks.
• The method provides a scalable solution for complex visual recognition and generation tasks.
• External attention facilitates the development of efficient deep learning architectures with comparable or improved performance.