Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning.

Area of Science:

• Neuroscience
• Computer Vision
• Machine Learning

Background:

• Brain decoding aims to reconstruct perceived stimuli from brain activity, such as functional Magnetic Resonance Imaging (fMRI) data.
• Significant challenges include the domain gap between fMRI signals and visual images, and limitations of fMRI data (low SNR, high dimensionality, low resolution).

Purpose of the Study:

• To develop a novel framework for reconstructing visual stimuli from fMRI signals by learning latent cognitive representations.
• To address the domain gap and data limitations inherent in fMRI-based brain decoding.

Main Methods:

• Introduced the Dual-Variational Autoencoder/Generative Adversarial Network (D-Vae/Gan) framework.
• Combined adversarial representation learning with knowledge distillation.
• Implemented a novel three-stage learning strategy for cognitive encoder knowledge distillation.

Main Results:

• The D-Vae/Gan framework demonstrated promising image reconstruction from fMRI signals.
• Achieved superior performance compared to existing methods on both artificial and natural images.
• Successfully learned visually-guided latent cognitive representations from fMRI data.

Conclusions:

• The proposed D-Vae/Gan method offers a viable solution for accurate image reconstruction from fMRI data.
• The approach effectively overcomes the domain gap and data limitations in brain decoding.
• This work advances the field of brain-computer interfaces and cognitive neuroscience.