A self-supervised deep neural network for image completion resembles early visual cortex fMRI activity patterns for...

Area of Science:

• Neuroscience
• Artificial Intelligence
• Computer Vision

Background:

• Artificial intelligence (AI) models, particularly convolutional neural networks (CNNs), have advanced the understanding of visual information processing by mimicking the brain's feedforward pathways.
• However, their ability to model feedback processes in early visual cortex remains less understood.

Purpose of the Study:

• To investigate the similarity between human early visual cortex activity and an AI model with an encoder/decoder architecture trained using self-supervised learning for image completion.
• To compare this self-supervised model against a traditional supervised object-recognition network (VGG16) in explaining brain data.

Main Methods:

• Utilized representational similarity analysis (RSA) to compare functional magnetic resonance imaging (fMRI) data from human participants viewing partially occluded images with the activation layers of a CNN.
• The CNN was trained with self-supervised learning to reconstruct occluded image regions.

Main Results:

• The self-supervised image-completion CNN demonstrated higher similarity to human early visual cortex fMRI data compared to the supervised VGG16 network.
• Activations in the decoder pathway of the CNN showed greater similarity to brain processing than encoder pathway activations.
• This suggests an integration of mid- and low/middle-level features within early visual cortex.

Conclusions:

• Findings support the hypothesis that AI models with less feedforward architectures and less supervised training may offer superior insights into biological visual processing.
• Comparing AI models trained via self-supervised learning with brain data aids in understanding complex information processing, including neuronal predictive coding mechanisms.