Generative Adversarial Networks Conditioned on Brain Activity Reconstruct Seen Images

Area of Science:

• Neuroscience
• Computer Vision
• Machine Learning

Background:

• Reconstructing visual stimuli from brain activity is challenging due to noisy, high-dimensional neural representations.
• Existing methods require strong prior knowledge to overcome incomplete information in neural data.

Purpose of the Study:

• To train generative adversarial networks (GANs) for image reconstruction conditioned on brain activity measurements.
• To address data limitations and noise in functional Magnetic Resonance Imaging (fMRI) data for improved generative modeling.

Main Methods:

• Developed a conditional generative model using GANs trained on surrogate brain activity samples.
• Employed an encoding model to generate surrogate brain activity data for GAN training.
• Validated the generative model on real fMRI data acquired during visual stimulus perception.

Main Results:

• The trained GAN model successfully generalized to real fMRI data.
• The model was able to reconstruct the basic outlines of perceived visual stimuli from brain activity.
• The strategy of using surrogate brain activity samples addressed data scarcity and noise robustness.

Conclusions:

• Conditional GANs, trained with surrogate data, offer a viable approach for reconstructing visual stimuli from brain activity.
• This method shows promise for advancing brain-computer interfaces and understanding neural representations of vision.
• Future work could refine reconstructions for greater detail and explore applications in other sensory modalities.