Nyströmformer: A Nystöm-based Algorithm for Approximating Self-Attention.

Area of Science:

• Natural Language Processing
• Machine Learning
• Artificial Intelligence

Background:

• Transformers excel in NLP due to self-attention mechanisms.
• Self-attention's quadratic complexity limits processing of long sequences.
• Efficient Transformer variants are crucial for advancing NLP applications.

Purpose of the Study:

• Introduce Nyströmformer, a scalable Transformer model.
• Address the quadratic complexity bottleneck of standard self-attention.
• Enable efficient processing of sequences with thousands of tokens.

Main Methods:

• Adapt the Nyström method to approximate self-attention.
• Achieve linear O(n) computational complexity with respect to sequence length.
• Evaluate Nyströmformer on GLUE, IMDB, and Long Range Arena benchmarks.

Main Results:

• Nyströmformer demonstrates comparable or superior performance to standard self-attention on GLUE and IMDB.
• Achieves favorable results on long-range sequence tasks in the LRA benchmark.
• Outperforms other efficient self-attention methods on long sequences.

Conclusions:

• Nyströmformer offers a scalable and efficient solution for Transformer models.
• Enables effective processing of extended text data.
• Paves the way for advanced NLP applications on longer sequences.