Reconstructing faces from fMRI patterns using deep generative neural networks.

Area of Science:

• Neuroscience
• Artificial Intelligence
• Computer Vision

Background:

• Decoding distinct categories from fMRI is reliable, but distinguishing visually similar stimuli like faces remains challenging.
• Deep learning models, particularly variational auto-encoders (VAEs) and generative adversarial networks (GANs), offer powerful tools for image analysis and reconstruction.

Purpose of the Study:

• To develop and apply a deep learning system for reconstructing face images from human functional Magnetic Resonance Imaging (fMRI) data.
• To investigate the feasibility of decoding visually similar inputs, specifically different faces, using fMRI and advanced AI.

Main Methods:

• A variational auto-encoder (VAE) neural network was trained using a Generative Adversarial Network (GAN) unsupervised procedure on a large dataset of celebrity faces.
• A simple linear mapping was established between multi-voxel fMRI activation patterns and the 1024 latent dimensions of the VAE's latent space.
• The learned mapping was applied to novel test images, translating fMRI patterns into VAE latent codes and subsequently reconstructing face images.

Main Results:

• The system achieved robust pairwise decoding of faces with >95% accuracy.
• Accurate gender classification was performed based on reconstructed faces.
• The system successfully decoded which face was imagined by subjects, not just seen.

Conclusions:

• Deep learning, combined with fMRI, enables high-fidelity reconstruction and decoding of complex visual stimuli like faces.
• This approach significantly advances the ability to interpret brain activity related to visual perception and imagination.
• The developed system holds promise for future brain-computer interface applications and understanding visual processing in the brain.