NeuroFusion: A Unified Framework for Generalized Visual Stimulus Decoding from fMRI Across Datasets and Subjects

Area of Science:

• Neuroscience
• Computer Vision
• Machine Learning

Background:

• Neural decoding aims to reconstruct sensory experiences from brain activity.
• Current methods struggle with generalization across different datasets and subjects due to variability.
• Functional Magnetic Resonance Imaging (fMRI) is a key neuroimaging technique for studying brain activity.

Purpose of the Study:

• To develop a unified framework for decoding visual stimuli across subjects and fMRI datasets.
• To address intersubject and interdataset variability in neural decoding.
• To enable robust generalization of visual decoding models.

Main Methods:

• Proposed a novel framework integrating multiple public fMRI datasets.
• Introduced a contrastive learning-based alignment strategy using IP-Adapter image embeddings.
• Developed a data augmentation method using ridge regression to synthesize realistic fMRI signals.

Main Results:

• Achieved strong semantic reconstruction across datasets, demonstrating robust cross-subject and cross-dataset transfer.
• Reported high CLIP similarity (up to 94.8%) and SSIM (0.403) on benchmark datasets after fine-tuning.
• Validated the effectiveness of the unified training framework across diverse fMRI data.

Conclusions:

• The proposed framework enables generalized visual decoding across subjects and datasets.
• The contrastive learning and data augmentation strategies effectively handle neural data variability.
• This work advances the field of neural decoding by enabling cross-dataset transfer learning.