Short-term Hebbian learning can implement transformer-like attention

Area of Science:

• Computational Neuroscience
• Machine Learning
• Artificial Intelligence

Background:

• Transformers have significantly advanced machine learning in language and vision.
• The neurobiological underpinnings of transformer attention mechanisms remain largely unexplored.
• Traditional neural circuits face challenges in implementing the mass comparisons required by attention layers.

Purpose of the Study:

• To propose a neurobiologically plausible mechanism for attention-like computations in neurons.
• To bridge the gap between transformer models and neuroscience.
• To introduce the match-and-control principle for neural attention.

Main Methods:

• Modeling neurons implementing attention-like computations.
• Utilizing short-term, Hebbian synaptic potentiation as the core mechanism.
• Representing keys and queries as spike trains for comparison within individual dendritic spines.

Main Results:

• Demonstrated that neurons can perform attention-like computations via the match-and-control principle.
• Showcased how synchronous activity (match) leads to transient synaptic potentiation (control).
• Indicated that individual spines can perform numerous key-query comparisons efficiently.

Conclusions:

• The match-and-control principle offers a novel framework for understanding neural computation.
• This mechanism provides a potential biological basis for attention in neural networks.
• The findings suggest a more direct link between artificial intelligence and neuroscience.