Aligning machines and minds: neural encoding for high-level visual cortices based on image captioning task.

Area of Science:

• Neuroscience
• Computer Vision
• Artificial Intelligence

Background:

• Neural encoding models predict brain responses to visual stimuli.
• Deep neural networks excel in early visual areas but struggle with high-level semantic representations.
• Limited encoding performance and interpretability exist for complex visual information.

Purpose of the Study:

• To develop a novel neural encoding model guided by image captioning for high-level visual cortices.
• To improve the encoding performance and interpretability of neural responses to complex visual stimuli.
• To bridge the domain gap between vision-language tasks and voxel-wise brain activity patterns.

Main Methods:

• Proposed a neural encoding model guided by image captioning.
• Utilized an attention module to focus on key visual objects during image captioning.
• Designed a flexible receptive field (RF) module to simulate voxel-level visual fields.
• Introduced the Atten-RF module to align attention-guided representations with brain activity.

Main Results:

• Achieved superior average encoding performance across seven high-level visual cortices.
• Reported a mean squared error of 0.765, Pearson correlation coefficient of 0.443, and R-squared of 0.245.
• Demonstrated enhanced prediction of voxel activity using a large-scale natural scenes dataset.

Conclusions:

• Leveraging vision-language tasks enhances neural encoding in high-level visual cortex.
• The Atten-RF module offers a new perspective on neural encoding.
• Visualization techniques provide insights into neural mechanisms of visual processing.