NeuroFusionNet: cross-modal modeling from brain activity to visual understanding

Area of Science:

• Neuroscience
• Computer Vision
• Machine Learning

Background:

• The integration of machine vision and neuroscience offers novel approaches to understanding visual information processing.
• Deep learning models are increasingly utilized to analyze complex data from both visual stimuli and neural activity.

Purpose of the Study:

• To propose NeuroFusionNet, an innovative deep learning model for enhanced visual information understanding.
• To integrate functional magnetic resonance imaging (fMRI) signals with image features for improved machine vision capabilities.

Main Methods:

• Images are processed to extract region-of-interest (ROI) features and contextual information.
• fMRI signals are processed using 1D convolutional layers and embedded into a 3D voxel representation.
• A Multi-scale fMRI Timeformer module and an fMRI-guided loss function are introduced for optimized performance.

Main Results:

• NeuroFusionNet effectively integrates image features and brain activity data.
• The model provides more precise and richer visual representations for machine vision systems.
• Experimental results demonstrate the model's capability in understanding visual information.

Conclusions:

• NeuroFusionNet represents a significant advancement in integrating neuroimaging data with computer vision.
• The developed model holds broad potential for applications requiring sophisticated visual understanding.
• This approach paves the way for more biologically plausible and effective machine vision systems.