Reconstructing shared visual experiences from human brain activity across individuals.

Area of Science:

• Neuroscience
• Computer Science
• Artificial Intelligence

Background:

• Reconstructing visual experiences from brain activity is key for advancing brain-computer interfaces and understanding perception.
• Current functional magnetic resonance imaging (fMRI)-based image synthesis methods are often person-specific, limiting scalability due to extensive data requirements for new individuals.

Purpose of the Study:

• To introduce MindShow, a unified generative framework for shared-subject fMRI-to-image reconstruction using cohort-level training.
• To enable data-efficient adaptation to new subjects with limited calibration data.

Main Methods:

• Developed a Hierarchically-Conditioned Mixture-of-Experts (HiCo-MoE) encoder to disentangle population-shared and subject-specific neural representations.
• Utilized a Gated Perceiver Bottleneck (GPB) to map fMRI features into fixed-size image and text latent tokens, addressing multi-scale representational misalignment.
• Introduced a multi-granular optimal transport loss (MOT-Align) for semantic and structural consistency by aligning brain-derived features with a pre-trained vision-language model.
• Employed a frozen diffusion model guided by aligned embeddings to synthesize images preserving semantic content and coarse layout.

Main Results:

• MindShow demonstrates improved high-level reconstruction metrics compared to existing methods.
• The framework maintains competitive structural fidelity in synthesized images.
• Achieved scalable shared-subject neural decoding with data-efficient adaptation.

Conclusions:

• MindShow offers a significant methodological advancement for scalable neural decoding from fMRI data.
• The framework facilitates the reconstruction of visual experiences across subjects more efficiently.
• This approach holds promise for broader applications in brain-computer interfaces and cognitive neuroscience research.