Talking to the Brain: Using Large Language Models as Proxies to Model Brain Semantic Features.

Area of Science:

• Neuroimaging
• Cognitive Neuroscience
• Artificial Intelligence

Background:

• Neuroimaging research faces challenges in analyzing naturalistic stimuli due to complex annotation and high-dimensional data.
• Manual annotation of complex visual scenes is time-consuming and limits the scale of neuroimaging studies.

Purpose of the Study:

• To develop a novel, scalable method for analyzing brain activity elicited by naturalistic stimuli.
• To leverage multimodal large language models (LLMs) for automated semantic feature extraction and brain activity prediction.
• To validate the methodology by replicating known neural correlates and exploring novel semantic representations in the brain.

Main Methods:

• Utilized multimodal large language models (LLMs) with a Visual Question Answering (VQA) strategy to process complex visual scenes.
• Transformed visual scenes into structured semantic feature vectors for predicting voxel-wise brain activity.
• Validated the model by replicating neural correlates for categories like faces and buildings.
• Mapped cortical activation patterns for 80 diverse semantic labels.

Main Results:

• The LLM-based approach successfully replicated established neural correlates for object categories.
• Cortical activation patterns for 80 semantic labels were mapped, revealing functional organization.
• A data-driven brain semantic similarity space was constructed, showing clusters related to functional and contextual associations.
• The method demonstrated scalability and effectiveness in analyzing brain responses to naturalistic stimuli.

Conclusions:

• Multimodal LLMs offer a high-throughput, automated solution for semantic annotation in neuroimaging.
• This "talking to the brain" approach enables ecologically valid investigations of brain semantic organization.
• The methodology overcomes limitations of manual annotation, facilitating large-scale studies of human cognition.