Deep Neural Encoder-Decoder Model to Relate fMRI Brain Activity with Naturalistic Stimuli

Area of Science:

• Neuroscience
• Artificial Intelligence
• Computer Vision

Background:

• Functional magnetic resonance imaging (fMRI) captures brain activity but has temporal resolution limitations.
• Naturalistic stimuli like movies offer rich visual information for studying brain function.
• Bridging the gap between stimulus presentation and neural recording is crucial for accurate brain decoding.

Purpose of the Study:

• To develop an end-to-end deep neural encoder-decoder model for encoding and decoding brain activity from fMRI data.
• To reconstruct visual stimuli from neural activity and identify brain regions involved in visual processing.
• To leverage deep learning models as a proxy for understanding visual processing in the brain.

Main Methods:

• An end-to-end deep neural encoder-decoder model was proposed.
• Temporal convolutional layers were employed to handle the temporal resolution gap between movie stimuli and fMRI.
• Saliency maps were used to investigate brain regions contributing to visual decoding.

Main Results:

• The model successfully predicted voxel activity in and around the visual cortex.
• Reconstruction of visual inputs from neural activity was achieved, including edges, faces, and contrasts.
• Key brain regions identified include the middle occipital area (shape perception), fusiform area (complex recognition, faces), and calcarine (basic visual features).

Conclusions:

• Deep learning models can effectively decode brain activity related to visual stimuli from fMRI data.
• The model's ability to reconstruct visual features aligns with the known functions of identified brain regions.
• Deep learning models serve as a valuable proxy for probing and understanding visual processing in the brain during naturalistic viewing.