From Questions to Neural Insights: Towards Query-Based fMRI Decoding.

Area of Science:

• Neuroscience
• Artificial Intelligence
• Computer Vision

Background:

• Decoding brain activity for visual and semantic representations is a key neuroscience challenge.
• Existing methods struggle to capture the complexity of neural representations.

Purpose of the Study:

• To introduce a novel multimodal framework for decoding functional magnetic resonance imaging (fMRI) data.
• To translate neural signals into textual descriptions and guide visual reconstruction.
• To enable fine-grained neural feature analysis by aligning brain activity with a shared visual-textual space.

Main Methods:

• Integration of BLIP-2 (vision-language model) and Stable Diffusion.
• Translation of neural signals into structured textual descriptions.
• Alignment of fMRI-derived embeddings within a shared visual-textual space.
• Analysis of hierarchical information from low-level perception to high-level semantics.

Main Results:

• Demonstrated efficacy in preserving key visual attributes within neural representations.
• Successful capture of hierarchical information, from perceptual to semantic levels.
• Validation of the framework's potential for advancing fMRI-based multimodal decoding.

Conclusions:

• The framework offers deeper insights into neural mechanisms of visual perception.
• It represents a step towards tools for decoding and reconstructing mental imagery.
• Potential applications include neurological diagnostics, cognitive decline detection, and brain-to-text interfaces.