High-level visual representations in the human brain are aligned with large language models.

Area of Science:

• Cognitive Neuroscience
• Artificial Intelligence
• Computer Vision

Background:

• The human brain processes intricate visual information from natural scenes, including object relations and environmental interactions.
• A quantitative method to study this complex visual information processing in the brain is currently lacking.

Purpose of the Study:

• To investigate if contextual information from large language models (LLMs) can aid in modeling the brain's extraction of complex visual information from natural scenes.
• To determine if LLM embeddings of scene captions can characterize and predict brain activity patterns.

Main Methods:

• LLM embeddings of scene captions were used to model brain activity evoked by viewing natural scenes.
• Model comparisons were conducted to assess the contribution of LLMs' integrated information processing.
• Deep neural networks were trained to map image inputs to LLM representations.

Main Results:

• LLM embeddings of scene captions successfully characterized brain activity, enabling accurate scene caption reconstruction from neural data.
• The accuracy of LLM-brain mapping stems from LLMs' ability to integrate complex information beyond individual words.
• Trained deep neural networks achieved superior alignment with brain representations compared to state-of-the-art models, despite limited training data.

Conclusions:

• LLM embeddings of scene captions offer a valuable representational format for understanding complex visual information processing in the brain.
• This approach provides a quantitative framework for studying visual cognition and its neural underpinnings.
• The findings highlight the potential of AI models in neuroscience research for decoding brain representations.