Transformer Mechanisms Mimic Frontostriatal Gating Operations When Trained on Human Working Memory Tasks

Area of Science:

• Cognitive Neuroscience
• Artificial Intelligence
• Computational Neuroscience

Background:

• Working memory (WM) relies on frontostriatal mechanisms for selective gating and information management.
• Computational models suggest WM capacity limits stem from difficulties in "role addressability" – binding items to roles and gating operations.
• Existing models are based on biological neural networks, leaving generalization to other architectures unclear.

Purpose of the Study:

• To investigate if gating mechanisms emerge in Transformer architectures, which lack built-in gating.
• To analyze Transformer mechanisms trained on human WM tasks demanding gating.
• To determine if Transformers can replicate frontostriatal-like gating principles.

Main Methods:

• Analyzing Transformer neural network architectures trained on specific working memory tasks.
• Examining the emergent properties of the Transformer's attention mechanism under gating demands.
• Comparing Transformer performance and mechanisms to computational models of frontostriatal function.

Main Results:

• Transformer attention mechanisms developed role-addressable input and output gating when trained on tasks benefiting from frontostriatal-like gating.
• These emergent gating strategies enhanced generalization and variable binding.
• The models showed increased effective capacity for storing and accessing multiple memory items.

Conclusions:

• Gating mechanisms play a fundamental computational role in managing role addressability and binding across different architectures.
• Transformer architectures can develop sophisticated gating strategies, mirroring biological systems.
• This research opens avenues for exploring computational similarities between AI and the human brain.