Revealing Vision-Language Integration in the Brain with Multimodal Networks

Area of Science:

• Neuroscience
• Artificial Intelligence
• Computational Neuroscience

Background:

• Understanding multimodal integration in the human brain is crucial for cognitive science.
• Deep neural networks (DNNs) offer powerful tools for analyzing complex neural data.
• Stereoencephalography (SEEG) provides high-resolution neural recordings for brain activity studies.

Purpose of the Study:

• To identify brain regions involved in multimodal (vision and language) integration using DNNs.
• To compare the predictive power of multimodal versus unimodal DNNs on SEEG recordings.
• To evaluate different DNN architectures and training techniques for predicting neural activity.

Main Methods:

• Utilized multimodal deep neural networks (DNNs) to predict SEEG recordings from subjects watching movies.
• Operationalized multimodal integration sites as regions where multimodal models outperform unimodal or linear models.
• Conducted controlled comparisons between models with identical architectures and training sets, varying only input modality.

Main Results:

• Trained vision and language DNNs significantly outperformed randomly initialized models in predicting SEEG signals.
• Identified a substantial number of neural sites (12.94% on average) and brain regions showing evidence of multimodal integration.
• CLIP-style training emerged as the most effective technique for downstream prediction of neural activity at these sites.

Conclusions:

• Multimodal DNNs are effective tools for discovering sites of multimodal integration in the human brain.
• Specific brain regions exhibit significant multimodal integration capabilities, as evidenced by improved DNN predictions.
• CLIP-style training demonstrates superior performance in modeling neural responses related to multimodal processing.