Sparse factorization of square matrices with application to neural attention modeling.

Area of Science:

• Machine Learning
• Matrix Approximation
• Deep Learning

Background:

• Large square matrices are computationally expensive in machine learning.
• Conventional low-rank approximations are inefficient for high-rank matrices.
• Scalable neural attention modeling requires efficient matrix approximation.

Purpose of the Study:

• To develop an economical approximation for large square matrices.
• To improve the efficiency of neural attention mechanisms.
• To address the limitations of low-rank matrix factorization.

Main Methods:

• Approximating large square matrices using a product of sparse full-rank matrices.
• Utilizing N(logN)^2 non-zero entries for an N×N matrix.
• Training neural networks to identify non-zero entries in factorizing matrices for attention modeling.

Main Results:

• The sparse factorization method provides superior approximation for sparse, high-rank matrices.
• The new attention module surpasses Transformer variants in performance on long sequences.
• Demonstrated effectiveness on synthetic datasets and Long Range Arena benchmarks.

Conclusions:

• Sparse full-rank matrix factorization is an efficient alternative to low-rank approximations.
• The proposed method enhances scalability and performance in neural attention models.
• This technique offers significant advantages for processing long sequences in deep learning.