BrainCLIP: Brain Representation via CLIP for Generic Natural Visual Stimulus Decoding

Area of Science:

• Neuroscience
• Computer Vision
• Machine Learning

Background:

• Functional Magnetic Resonance Imaging (fMRI) faces limitations like small sample sizes and low signal-to-noise ratios.
• Challenges exist in reconstructing natural images and decoding semantic content from fMRI data.

Purpose of the Study:

• Introduce BrainCLIP, an fMRI-based brain decoding model.
• Leverage Contrastive Language-Image Pre-training (CLIP) for cross-modal generalization between brain activity, images, and text.
• Enhance fMRI decoding capabilities.

Main Methods:

• Developed BrainCLIP, integrating CLIP's cross-modal abilities.
• Trained a mapping network to translate fMRI patterns into a unified CLIP embedding space.
• Utilized integrated visual and textual supervision.

Main Results:

• Demonstrated BrainCLIP's effectiveness in zero-shot visual category decoding, fMRI-image/text alignment, and fMRI-to-image generation.
• Achieved significant performance improvements in fMRI-text alignment and image generation.
• Outperformed the BraVL multi-modal method in zero-shot visual category decoding.
• Showcased high semantic fidelity in reconstructing visual stimuli from fMRI data.

Conclusions:

• BrainCLIP effectively bridges brain activity, images, and text using CLIP's cross-modal generalization.
• The model offers a powerful new tool for advanced fMRI-based brain decoding.
• BrainCLIP shows promise for future research in neuroimaging and artificial intelligence integration.