Cross-subject fMRI-to-Image with Visual-cortex 2D Representation and Pre-Training

Decoding brain activity from functional Magnetic Resonance Imaging (fMRI) data is crucial in neuro science, yet traditional methods have largely focused on analyzing data from individual subjects. In this paper, we present X-MinD, a novel zero-shot cross-subject fMRI-to image approach designed to reconstruct images from brain activity across unseen subjects. By converting fMRI visual cortex signals into 2D surface images, we offer a fresh perspective on learning representations of brain activity from structured visual textures instead of flattened response signals. Utilizing such an fMRI representation mitigates differences across multiple individuals. Importantly, it enables the development of a customized visual autoencoder for learning generalizable visual-cortex fMRI features through self-supervised pre-training. With a straightforward linear mapping between fMRI features and image feature space, our framework achieves effective zero-shot decoding of cross-subject fMRI activity into realistic and semantically consistent images, bypassing the need for model fine tuning. We conduct extensive experiments to establish the superior performance of our framework for zero-shot cross subject brain decoding.