Brain-aligning of semantic vectors improves neural decoding of visual stimuli

Area of Science:

• Neuroscience
• Machine Learning
• Computational Neuroscience

Background:

• Brain decoding aims to map neural data to stimulus features using machine learning.
• Current methods use image/text feature vectors that may not align with neural encoding.
• This misalignment limits the accuracy of brain decoding algorithms.

Purpose of the Study:

• To introduce a framework, brain-aligning of semantic vectors, to fine-tune feature vectors for better alignment with neural representations.
• To improve the accuracy of brain decoding by bridging the gap between artificial feature spaces and neural encoding.

Main Methods:

• Developed a framework to fine-tune pretrained feature vectors to align with neural representations.
• Trained the model using functional magnetic resonance imaging (fMRI) data.
• Evaluated zero-shot brain decoding performance on fMRI, magnetoencephalography (MEG), and electrocorticography (ECoG) data.

Main Results:

• Brain-aligned vectors derived from fMRI improved decoding performance across fMRI, MEG, and ECoG datasets.
• Accuracy improvements were measured using correlation coefficients between true and predicted vectors.
• Stimulus identification accuracy increased for specific categories, with variations based on the original vector space used for alignment.

Conclusions:

• The proposed brain-aligning framework enhances neural decoding accuracy across multiple neuroimaging modalities.
• Fine-tuning semantic vectors to match neural representations offers a promising approach for more effective brain decoding.
• This method demonstrates consistent improvements regardless of the neuroimaging technique used.