Continual low-rank scaled dot-product attention.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Transformers excel at sequence processing for static tasks.
• The Scaled Dot-product Attention in Transformers has high computational and memory demands, limiting its use in real-time stream data processing.
• Existing methods to reduce Transformer costs include low-rank approximations and sparsity.

Purpose of the Study:

• To develop an efficient formulation of Scaled Dot-product Attention suitable for continual inference.
• To address the computational and memory constraints of Transformers in stream data applications.

Main Methods:

• Introduced a novel formulation of Scaled Dot-product Attention utilizing the Nyström approximation.
• Developed a method for efficient continual inference with Transformers.

Main Results:

• The proposed Continual Scaled Dot-product Attention reduced operations by up to three orders of magnitude compared to original Transformers.
• Maintained predictive performance comparable to existing models in online tasks.

Conclusions:

• The Nyström approximation offers a computationally efficient alternative for Scaled Dot-product Attention in stream data scenarios.
• This approach makes Transformers more feasible for real-time applications with resource constraints.