End-to-End Deep Image Reconstruction From Human Brain Activity.

Area of Science:

• Neuroscience
• Computer Science
• Machine Learning

Background:

• Deep neural networks (DNNs) show promise for brain decoding and image reconstruction from fMRI data.
• Direct DNN training with fMRI data is challenging due to limited data size.
• Current methods use pre-trained DNNs as proxies for visual representations, decoding fMRI data with linear models.

Purpose of the Study:

• To develop and evaluate an end-to-end deep neural network model for direct image reconstruction from fMRI data.
• To investigate the feasibility of training DNNs directly with fMRI responses and corresponding stimulus images.
• To improve the accuracy of reconstructing visual stimuli from brain activity.

Main Methods:

• A generative adversarial network was trained directly with fMRI data and stimulus images.
• An additional feature loss term in high-level feature space was incorporated.
• The model was trained using up to 6,000 data samples and tested on independent datasets.

Main Results:

• The end-to-end model successfully reconstructed images from fMRI patterns, resembling test stimuli.
• Reconstruction accuracy improved with increased training data size.
• Ablation studies confirmed the critical role of feature loss in accurate reconstruction.

Conclusions:

• Directly training DNNs with fMRI data enables end-to-end image reconstruction.
• The proposed method establishes a direct mapping between brain activity and visual perception.
• Feature loss is crucial for achieving high-fidelity reconstructions from fMRI data.