A visual encoding model based on deep neural networks and transfer learning for brain activity measured by functional...

Area of Science:

• Neuroscience
• Computer Vision
• Machine Learning

Background:

• Accurately predicting visual responses is crucial for brain-machine interfaces.
• Current models often use hand-crafted or deep neural network (DNN) features with linear mapping.
• The sufficiency of linear mapping for maximizing prediction accuracy remains unclear.

Purpose of the Study:

• To develop a novel visual encoding framework for predicting cortical responses.
• To incorporate pre-trained DNN features and nonlinear mapping for improved accuracy.
• To enhance brain-machine interface capabilities through better neural signal prediction.

Main Methods:

• Constructed a new visual encoding framework using a pre-trained DNN (AlexNet).
• Employed transfer learning to integrate visual features from the DNN.
• Trained a nonlinear mapping from visual features to brain activity, replacing linear assumptions.

Main Results:

• The framework significantly predicted responses in over 20% of voxels in early visual areas (V1-LO).
• Achieved unprecedented prediction accuracy compared to conventional models.
• Demonstrated consistently higher accuracy in all early visual areas, particularly V4 and LO.

Conclusions:

• Proposed a novel framework utilizing pre-trained visual features and nonlinear mappings.
• The nonlinear approach enhances the prediction of brain activity from visual stimuli.
• This advancement offers a more effective method for visual encoding in brain-machine interfaces.