Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space.

Area of Science:

• Neuroscience
• Computer Vision
• Machine Learning

Background:

• Neural decoding aims to map brain activity to sensory stimuli.
• Existing methods often struggle with naturalistic stimuli and high-fidelity reconstructions.

Purpose of the Study:

• To introduce a novel experimental paradigm for neural decoding.
• To achieve HYPerrealistic reconstruction of PERception (HYPER) of faces from brain recordings.
• To integrate generative adversarial networks (GANs) into the neural decoding pipeline.

Main Methods:

• Developed a novel paradigm using well-controlled, naturalistic stimuli.
• Acquired fMRI data while participants viewed GAN-synthesized face images.
• Utilized GAN latent vectors as intermediate feature representations for decoding.

Main Results:

• GAN latent vectors effectively captured defining stimulus properties from fMRI data.
• Predicted latents were used for neural decoding and stimulus regeneration.
• Achieved the most accurate perception reconstructions to date.

Conclusions:

• GANs can be effectively integrated into neural decoding pipelines.
• Latent representations from GANs serve as powerful intermediate features for decoding brain activity.
• The HYPER method significantly advances the accuracy of reconstructing perceived stimuli from neural data.