Mapping the movie-watching brain with AI-derived semantics.

Area of Science:

• Neuroscience
• Artificial Intelligence
• Cognitive Science

Background:

• Naturalistic stimuli, like movies, are valuable for studying brain function.
• Linking complex movie content to brain activity remains challenging.
• Interpreting distributed brain responses requires robust analytical frameworks.

Purpose of the Study:

• To use a multimodal large language model (Gemini) as a semantic annotator for naturalistic stimuli.
• To bridge the gap between movie content, brain responses, and cognitive performance.
• To explore the utility of AI-derived semantic features in understanding neural processing.

Main Methods:

• Segmented movie data into clips and prompted Gemini for semantic ratings on 11 dimensions.
• Extracted clip-wise BOLD activation patterns from fMRI data in 360 cortical regions.
• Fitted linear regression models to predict brain responses using AI-derived semantic features.

Main Results:

• Gemini features predicted brain activity in association cortices (temporal, medial parietal, lateral frontal) but not unimodal regions.
• Feature-weight maps aligned with known functional specializations.
• Individual differences in resting-state connectivity and AI predictability showed an asymmetric relationship with brain systems.

Conclusions:

• Interpretable AI features offer a scalable framework for quantifying semantic predictability in naturalistic settings.
• AI models can serve as semantic references to probe human neural processing.
• AI-derived semantic predictability is linked to cognitive abilities in specific brain regions.