Retrieving and reconstructing conceptually similar images from fMRI with latent diffusion models and a neuro-inspired...

Area of Science:

• Computational Neuroscience
• Artificial Intelligence
• Cognitive Neuroscience

Background:

• Brain decoding aims to infer mental states from brain activity.
• Existing methods often lack focus on semantic and contextual understanding of stimuli.

Purpose of the Study:

• Propose a novel brain decoding approach leveraging semantic and contextual similarity.
• Develop a deep learning pipeline to map brain activity to semantic features of visual stimuli.

Main Methods:

• Utilized functional magnetic resonance imaging (fMRI) datasets with natural image stimuli.
• Created a linear brain-to-feature model mapping fMRI activity to semantic features from a pre-trained neural network.
• Employed nearest-neighbor strategies for feature categorization and image retrieval/generation using latent diffusion models.

Main Results:

• Demonstrated successful semantic classification and image retrieval across three fMRI datasets (Generic Object Decoding, BOLD5000, NSD).
• Achieved over 80% accuracy in human evaluation for image similarity assessment.
• Showcased the ability to generate novel images matching original visual content.

Conclusions:

• Measurable neural correlates can be linearly mapped to a neural network's latent space for image synthesis.
• Semantic and contextual similarity offers a powerful new direction for brain decoding.
• Findings have significant implications for understanding visual perception and advancing AI.