Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision.

Area of Science:

• Neuroscience
• Computer Vision
• Machine Learning

Background:

• Convolutional neural networks (CNNs) explain cortical responses to static images in the ventral stream.
• Current models lack mechanisms for temporal dynamics or feedback processing.

Purpose of the Study:

• To investigate CNNs' ability to predict and decode functional magnetic resonance imaging (fMRI) data from humans viewing natural movies.
• To explore the bidirectional relationships between CNNs and brain activity.

Main Methods:

• Developed and evaluated encoding and decoding models using separate datasets.
• Utilized CNNs to predict brain activity and fMRI signals.
• Visualized single-voxel responses and synthesized cortical activation.

Main Results:

• CNN predictions extended beyond the ventral stream to the dorsal stream.
• Individual cortical locations showed distinct representations.
• fMRI signals were decoded to estimate feature representations for visual reconstruction and semantic categorization.

Conclusions:

• CNNs serve as a comprehensive model for the visual cortex.
• Deep learning effectively decodes natural vision and generalizes previous findings.
• This approach advances understanding of visual processing in the brain.