NeuroFusion: A Unified Framework for Generalized Visual Stimulus Decoding from fMRI Across Datasets and Subjects

Area of Science:

• Neuroscience
• Computer Vision
• Machine Learning

Background:

• Neural decoding aims to reconstruct sensory experiences from brain activity.
• Current methods struggle with generalization across different datasets and subjects due to variability.
• Limited cross-dataset transfer hinders the broad application of neural decoding models.

Purpose of the Study:

• To develop a novel framework for decoding visual stimuli across subjects and datasets.
• To address intersubject and interdataset variability in functional Magnetic Resonance Imaging (fMRI) data.
• To enable generalized visual decoding by integrating multiple fMRI datasets.

Main Methods:

• A contrastive learning-based alignment strategy using pre-trained IP-Adapter image embeddings.
• Learning a shared latent space by aligning neural representations with image features.
• A data augmentation method using ridge regression to synthesize realistic fMRI-like signals.

Main Results:

• Achieved strong semantic reconstruction across datasets, demonstrating robust cross-subject and cross-dataset transfer.
• Validated performance with high CLIP similarity (94.8%) on NSD (AUG) and SSIM (0.403) on BOLD5000.
• The unified framework trained across multiple fMRI datasets showed significant generalization capabilities.

Conclusions:

• The proposed framework enables generalized neural decoding across subjects and datasets.
• The integration of contrastive learning and data augmentation enhances model robustness and transferability.
• This work advances cross-dataset transfer in neural decoding, paving the way for more versatile applications.