Inter-individual deep image reconstruction via hierarchical neural code conversion.

Area of Science:

• Neuroscience
• Cognitive Science
• Machine Learning

Background:

• Sensory cortex exhibits organizational principles like topography and hierarchy.
• Individual differences in brain activity patterns persist despite identical sensory input.
• Existing functional magnetic resonance imaging (fMRI) alignment methods have limitations in preserving hierarchical and fine-grained representations across individuals.

Purpose of the Study:

• To develop and evaluate a functional alignment method, the neural code converter, for transferring brain activity patterns between individuals.
• To assess whether hierarchical and fine-grained representations can be converted while preserving encoded perceptual content.
• To demonstrate the capability of converting neural representations for inter-individual visual image reconstruction.

Main Methods:

• Trained a neural code converter on fMRI responses from pairs of individuals viewing identical natural images.
• Utilized visual cortex voxels from V1 through ventral object areas without explicit visual area labels.
• Decoded converted brain activity patterns into hierarchical visual features of a deep neural network (DNN) using pre-trained decoders.
• Reconstructed perceived images from decoded features.

Main Results:

• The neural code converter automatically learned correspondences between visual areas of corresponding hierarchical levels.
• DNN feature decoding accuracy was higher at corresponding visual area levels, confirming preserved hierarchical representations.
• Image reconstruction yielded recognizable object silhouettes, even with limited training data.
• Decoders trained on pooled data via conversions showed slight improvement over single-individual training.

Conclusions:

• Functional alignment using the neural code converter effectively transfers hierarchical and fine-grained neural representations between individuals.
• The conversion process preserves sufficient visual information for inter-individual image reconstruction.
• This approach advances our understanding of cross-individual neural coding and has implications for brain data analysis and sharing.