State-space modeling in long sequence processing: a survey on recurrence in the transformer era.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Learning from sequential data, especially long sequences, is a core AI challenge.
• Traditional methods and Transformers have limitations in capturing long-term dependencies.
• Recurrent Neural Networks (RNNs) are experiencing a resurgence due to new State-Space Models and large-context Transformers.

Purpose of the Study:

• To provide an in-depth summary of recent recurrent model-based approaches for sequential data processing.
• To offer a taxonomy of current trends in recurrent architectures and algorithms.
• To guide researchers in the field of sequential data processing.

Main Methods:

• Survey of recent literature on recurrent models for sequential data.
• Analysis of architectural and algorithmic innovations.
• Discussion of emerging trends and their implications.

Main Results:

• Recurrent computations are key to overcoming limitations of existing technologies like Transformers.
• Deep State-Space Models and large-context Transformers demonstrate the revival of recurrent approaches.
• Novel recurrent architectures offer improved efficiency for processing long sequences.

Conclusions:

• The field is moving towards more realistic online processing of sequential data.
• There is potential for developing learning algorithms beyond Backpropagation Through Time.
• Future research can focus on local-forward computations for efficient, real-time sequential data processing.