Transformers and cortical waves: encoders for pulling in context across time.

Area of Science:

• Computational neuroscience
• Artificial intelligence
• Cognitive science

Background:

• Large language models (LLMs) like ChatGPT excel at processing sequential data using transformer networks.
• Transformers rely on self-attention mechanisms to capture long-range temporal dependencies within input sequences.
• Understanding the neural basis of temporal information processing in the brain is a key challenge.

Purpose of the Study:

• To propose a neurobiological mechanism for temporal sequence processing analogous to transformer networks.
• To explore how cortical waves might implement a similar computational principle as self-attention in LLMs.

Main Methods:

• Theoretical modeling comparing transformer computation with neural activity patterns.
• Conceptual analysis of self-attention mechanisms and their potential neural correlates.
• Hypothesizing the role of traveling waves in neural information encoding.

Main Results:

• Transformer networks encode input sequences into vectors, enabling the capture of temporal dependencies.
• Self-attention within transformers computes pairwise word associations to enhance context.
• Cortical waves are proposed to encapsulate input history into spatial patterns, enabling temporal context extraction.

Conclusions:

• Cortical waves may offer a biological implementation of the encoding principles used in transformer networks.
• This neural mechanism could explain how the brain processes sequential sensory information.
• The study provides a novel perspective linking artificial intelligence computation to brain function.