Transformer brain encoders explain human high-level visual responses

Area of Science:

• Neuroscience
• Computational Neuroscience
• Computer Vision

Background:

• Understanding brain computations in naturalistic visual processing is a key neuroscience goal.
• Current methods using deep neural networks and linear encoding models have limitations in capturing feature map structures and dynamic routing.
• Previous alternatives focusing on static receptive fields are insufficient for high-level visual areas.

Purpose of the Study:

• To investigate how visual features are dynamically routed to category-selective brain areas using attention mechanisms.
• To develop a more powerful and interpretable computational model for visual processing in the human brain.
• To compare the efficacy of attention-based routing against existing encoding models.

Main Methods:

• Employed the attention mechanism from transformer architectures to model feature routing.
• Utilized image-computable deep neural networks as feature extractors.
• Tested the model's predictive power on brain activity data during natural scene viewing across various feature bases and modalities.

Main Results:

• The attention-based routing model significantly outperformed alternative methods in predicting brain activity.
• The model demonstrated superior performance across different feature basis models and modalities.
• Attention-routing signals were found to be easily visualizable, enhancing model interpretability.

Conclusions:

• Attention mechanisms provide a powerful computational motif for understanding dynamic visual feature routing in the brain.
• This approach offers a more mechanistic and interpretable model for high-level visual processing compared to existing methods.
• The model's high performance suggests its potential as a candidate for explaining how visual information is routed based on content relevance in the human brain.