Self-Attention-Based Contextual Modulation Improves Neural System Identification

Area of Science:

• Neuroscience
• Computational Neuroscience
• Artificial Intelligence

Background:

• Primary visual cortex neurons process contextual information via horizontal and feedback connections.
• Standard Convolutional Neural Networks (CNNs) use successive convolutions and readout layers for contextual modulation.
• Existing models struggle to fully capture the nuances of neural responses to contextual stimuli.

Purpose of the Study:

• To evaluate the efficacy of self-attention (SA) mechanisms in improving neural response predictions.
• To introduce and utilize 'peak tuning' as a metric for assessing a model's ability to capture a neuron's top feature preference.
• To investigate the complementary roles of local receptive fields and surround information in neural tuning.

Main Methods:

• Compared parameter-matched CNNs with SA models on neural response prediction tasks.
• Introduced 'peak tuning' as a novel evaluation metric.
• Factorized network components to isolate the contributions of different contextual mechanisms (local receptive field vs. surround information).

Main Results:

• Self-attention (SA) significantly improved neural response predictions over CNNs in terms of tuning curve correlation and peak tuning.
• Local receptive field information is vital for overall tuning, while surround information is critical for characterizing the tuning peak.
• SA can replace spatial-integration convolutions and is enhanced by fully connected readout layers, indicating complementary functions.

Conclusions:

• Self-attention mechanisms offer a powerful approach to modeling contextual modulation in visual cortical neurons.
• Understanding the interplay between local and surround information is key to accurately predicting neural responses.
• Incremental learning of receptive field and contextual modulation, particularly surround-center interactions, shows promise for robust neural modeling.