How does the brain represent the semantic content of an image?

Area of Science:

• Cognitive Neuroscience
• Computational Neuroscience
• Artificial Intelligence

Background:

• Deep neural networks (DNNs) are increasingly used as models for the biological brain.
• The 'black box' nature of DNNs poses a challenge for understanding their biological relevance.
• Neural style transfer offers a method to interpret DNNs by focusing on meaningful deep features.

Purpose of the Study:

• To investigate the biological interpretability of DNNs by examining deep features.
• To determine if deep features representing semantic image content predict brain activity.
• To explore the nature of visual representations in early visual areas.

Main Methods:

• Utilized deep features derived from DNNs, specifically those representing semantic image content.
• Employed encoding models to link deep features with brain activity.
• Applied representational similarity analysis to quantitatively compare DNN features and neural representations.

Main Results:

• Deep features encoding semantic image content significantly predicted voxel activity in early visual areas (V1, V2, V3).
• These semantic features were found to be both depictive and propositional in nature.
• The findings suggest a link between DNN representations and neural processing in the visual cortex.

Conclusions:

• Deep features, when interpretable as semantic content, can serve as valuable probes for understanding brain function.
• The results align with grounded cognition theories, indicating that visual representations are depictive and support symbolic functions.
• This approach offers a method to bridge the gap between artificial and biological neural networks.