Brain encoding models based on multimodal transformers can transfer across language and vision.

Area of Science:

• Neuroscience
• Cognitive Science
• Artificial Intelligence

Background:

• Encoding models analyze brain representations of concepts in language and vision.
• Current models typically process each modality separately, limiting cross-modal insights.
• Multimodal transformers offer aligned concept representations across language and vision.

Purpose of the Study:

• To train cross-modal encoding models using multimodal transformer representations.
• To investigate the transferability of encoding models between fMRI responses to stories and movies.
• To identify shared semantic dimensions in human concept representation.

Main Methods:

• Utilized representations from multimodal transformers to train brain encoding models.
• Tested model performance on predicting fMRI responses to both stories (language) and movies (vision).
• Analyzed encoding models to uncover shared semantic dimensions underlying cross-modal concept representation.

Main Results:

• Encoding models trained on one modality successfully predicted brain responses in the other modality.
• High transferability was observed in cortical regions associated with conceptual meaning.
• Multimodal transformers yielded more aligned representations compared to unimodal transformers.

Conclusions:

• Multimodal transformers facilitate the development of effective cross-modal brain encoding models.
• The findings highlight shared semantic structures in the brain's processing of language and vision.
• This research advances our understanding of the neural basis of multimodal information processing.