Decoding naturalistic experiences from human brain activity via distributed representations of words.

Area of Science:

• Neuroscience
• Cognitive Science
• Computational Linguistics

Background:

• Natural visual scenes evoke diverse and complex perceptual experiences.
• Understanding how the brain represents these experiences is a key challenge in neuroscience.
• Existing methods often struggle with the variability and richness of naturalistic stimuli.

Purpose of the Study:

• To propose and validate a new framework for decoding perceptual experiences from brain activity.
• To leverage distributed word representations for a high-dimensional feature space of perception.
• To associate functional magnetic resonance imaging (fMRI) data with semantic content of visual scenes.

Main Methods:

• Functional magnetic resonance imaging (fMRI) was used to record brain activity while participants viewed natural movie scenes.
• A skip-gram model was employed to create distributed word embeddings, forming a high-dimensional feature space.
• A decoder was trained to link fMRI-measured brain activity to these distributed word representations.

Main Results:

• The decoder successfully estimated perceptual content, including nouns (objects), verbs (actions), and adjectives (impressions), from brain activity.
• The framework demonstrated the ability to decode novel words not present in the training data.
• Inter-individual variability in decoded content correlated with variability in human scene descriptions.

Conclusions:

• The proposed decoding framework effectively captures diverse perceptual experiences from naturalistic visual stimuli.
• This approach offers a powerful tool for understanding brain representations of complex scenes.
• The method has potential applications in various scientific and practical fields requiring perception decoding.