May I see what you see? Predicting visual features from neuronal activity.

Area of Science:

• Neuroscience
• Computer Vision
• Machine Learning

Background:

• Understanding brain responses to audiovisual stimuli is crucial for deciphering neuronal processes.
• Functional magnetic resonance imaging (fMRI) measures brain activity by detecting associated changes in blood flow.

Purpose of the Study:

• To reconstruct video frames from fMRI data.
• To demonstrate the model's ability to predict visual objects from fMRI signals.
• To investigate the relationship between brain activity and visual perception.

Main Methods:

• An autoencoder model was trained on video segments to create latent representations of video streams.
• A mapping was learned between fMRI responses and the corresponding latent video frame representations.
• fMRI data was used to compute latent vectors, which were then passed through a decoder to reconstruct predicted images.

Main Results:

• Representations derived from video frames and corresponding fMRI images showed high clustering.
• The model successfully predicted objects in video frames using only fMRI data.
• fMRI responses enabled the reconstruction of inputs, predicting the presence of faces.

Conclusions:

• The study successfully reconstructed visual information from fMRI data, bridging neuroscience and computer vision.
• This approach demonstrates the potential for decoding visual content directly from brain activity.
• The findings open avenues for advanced brain-computer interfaces and understanding visual processing.