Reconstructing controllable faces from brain activity with hierarchical multiview representations.

Area of Science:

• Neuroscience
• Computer Vision
• Machine Learning

Background:

• Reconstructing visual experiences from fMRI data is challenging, particularly for visually similar stimuli like faces.
• Existing methods often struggle with precise facial attribute decoding, leading to indistinguishable reconstructed faces.

Purpose of the Study:

• To propose a novel neural decoding framework, VSPnet (voxel2style2pixel), for more elaborate visual stimuli reconstruction from fMRI.
• To improve the precision of decoding facial attributes in perceived face reconstruction.

Main Methods:

• VSPnet utilizes hierarchical encoding and decoding networks with disentangled latent representations.
• A hierarchical visual encoder (HVE) pre-extracts features from stimuli.
• The framework includes a multi-branch cognitive encoder and a StyleGAN-inspired image generator.

Main Results:

• VSPnet significantly outperforms state-of-the-art approaches in reconstruction accuracy.
• The identifiability of reconstructed faces is greatly improved.
• Feature editing for facial attributes in the fMRI domain was achieved using multi-view latent representations.

Conclusions:

• VSPnet offers a superior method for visual reconstruction from fMRI data, especially for faces.
• The framework's ability to disentangle representations enhances the detail and accuracy of reconstructed images.
• This work advances brain decoding by enabling precise attribute manipulation within the fMRI domain.