Limits to visual representational correspondence between convolutional neural networks and the human brain

Area of Science:

• Neuroscience
• Computer Science
• Artificial Intelligence

Background:

• Convolutional neural networks (CNNs) are widely used to model human vision due to their object recognition abilities and similarities to brain activity.
• Understanding the correspondence between CNNs and human visual processing is crucial for advancing both fields.

Purpose of the Study:

• To evaluate the performance of 14 different CNNs against human fMRI data for visual representation.
• To investigate CNNs' ability to model the brain's processing of natural and artificial images.

Main Methods:

• Representational Similarity Analysis (RSA) was employed to compare CNN model representations with human fMRI responses.
• 14 distinct CNN architectures were tested using natural and artificial image datasets.

Main Results:

• CNNs demonstrated a strong correspondence with lower-level visual representations of real-world objects.
• CNNs did not fully capture higher-level visual representations for either real-world or artificial objects.
• Results were consistent across different CNN architectures, training methods, and the inclusion of recurrent processing.

Conclusions:

• Despite successes in lower-level vision, CNNs exhibit fundamental differences from the human brain in representing complex visual information.
• Current CNN models do not adequately explain higher-level visual processing in the human brain for both natural and artificial stimuli.