Deep image reconstruction from human brain activity.

Area of Science:

• Neuroscience
• Computer Vision
• Cognitive Science

Background:

• Perception and imagery are encoded in hierarchical brain representations.
• Previous visualization methods struggled to use multiple hierarchical levels.
• Functional magnetic resonance imaging (fMRI) data can be decoded into deep neural network (DNN) features.

Purpose of the Study:

• To develop a novel image reconstruction method using multi-layer DNN features decoded from brain activity.
• To enhance the reconstruction of visual perception and mental imagery.
• To investigate the effectiveness of combining hierarchical neural representations.

Main Methods:

• Optimized image pixel values to match DNN features decoded from fMRI data across multiple layers.
• Utilized a deep generator neural network with a natural image prior for semantic detail.
• Applied the method to both visual perception and mental imagery tasks.

Main Results:

• Reliably reconstructed natural images that closely resembled the viewed stimuli.
• Incorporated semantically meaningful details into reconstructions via the natural image prior.
• Demonstrated rudimentary reconstructions of subjective mental imagery content.
• Showcased generalization to artificial shapes, not just trained exemplars.

Conclusions:

• The novel method effectively reconstructs perceptual and subjective images by integrating hierarchical neural representations.
• Combining multiple DNN layers significantly enhances the visual quality of reconstructed images.
• This approach offers a new method for exploring the brain's internal representational content.