A novel DRL-guided sparse voxel decoding model for reconstructing perceived images from brain activity.

Area of Science:

• Neuroscience
• Computer Vision
• Machine Learning

Background:

• Voxel selection is crucial for reconstructing images from fMRI due to sparse neural encoding and limited paired data.
• Existing data-driven voxel selection methods yield suboptimal image reconstruction results.

Purpose of the Study:

• To develop a novel deep reinforcement learning-guided sparse voxel (DRL-SV) decoding model for improved image reconstruction from fMRI.
• To address the limitations of current data-driven methods in voxel selection for visual encoding.

Main Methods:

• Voxel selection is framed as a Markov decision process (MDP).
• A deep reinforcement learning agent is trained to identify voxels critical for specific visual encoding.
• The DRL-SV model is evaluated on two public datasets.

Main Results:

• The DRL-SV model accurately identifies voxels highly involved in neural encoding.
• Experimental results demonstrate improved visual image reconstruction quality using DRL-SV.
• DRL-SV achieves superior reconstruction performance compared to traditional data-driven methods, with sparser voxel selection.

Conclusions:

• DRL-SV effectively selects crucial voxels for visual encoding, even with few-shot learning.
• The proposed decoding model offers a promising new approach for enhancing primary visual cortex image reconstruction quality.