Brain-optimized inference improves reconstructions of fMRI brain activity

Area of Science:

• Neuroscience
• Artificial Intelligence
• Computer Vision

Background:

• Recent advances in artificial intelligence (AI) and large datasets have significantly improved brain activity decoding for image reconstruction.
• Existing decoding methods reconstruct images from neural data but have room for improvement in fidelity and accuracy.

Approach:

• We developed a brain-optimized inference technique to refine image reconstructions.
• This method iteratively optimizes image candidates by aligning them with measured brain activity using an encoding model.
• A diffusion model generates image libraries, guided by previous reconstructions and constrained by brain data.

Key Points:

• The brain-optimized inference approach demonstrably outperforms baseline decoding methods.
• Improvements were validated through human ratings, image feature analysis, and neural alignment metrics.
• The refinement process shows systematic variations across the visual cortex, indicating diverse neural representations.

Conclusions:

• Explicitly aligning decoding output distributions with brain activity distributions enhances reconstruction quality.
• This method improves state-of-the-art decoding algorithms and offers insights into visual processing.
• Brain-optimized inference provides a novel approach for both enhancing image reconstruction and studying neural representations.