Semantics-Guided Hierarchical Feature Encoding Generative Adversarial Network for Visual Image Reconstruction From...

Area of Science:

• Neuroscience
• Computer Science
• Artificial Intelligence

Background:

• Decoding visual perception from fMRI data is challenging due to high dimensionality and low signal-to-noise ratio.
• Previous deep learning models for fMRI-based image reconstruction yield low-quality or unreliable results.
• Extracting meaningful visual information for perceptual reconstruction from fMRI is complex.

Purpose of the Study:

• To propose a novel neural decoding model, the hierarchical semantic generative adversarial network (HS-GAN).
• To leverage hierarchical and semantic representations for reconstructing perceptual images from fMRI data.
• To improve the naturalness and fidelity of reconstructed images from brain activity.

Main Methods:

• Developed the hierarchical semantic generative adversarial network (HS-GAN).
• Inspired by the hierarchical encoding of the visual cortex and convolutional neural network (CNN) homology theory.
• Utilized hierarchical and semantic representations for image reconstruction from fMRI data.

Main Results:

• HS-GAN achieved superior performance on the Horikawa2017 dataset compared to advanced methods.
• Demonstrated improved histogram similarity, SSIM-Acc, Perceptual-Acc, and AlexNet accuracy.
• Showcased versatility and generalization capabilities by reconstructing handwritten digits with high SSIM.

Conclusions:

• HS-GAN offers a significant advancement in reconstructing visual images from fMRI data.
• The model enhances the naturalness and fidelity of reconstructed perceptual images.
• HS-GAN shows potential for broader applications beyond natural image reconstruction.