Probing the content of semantic representations in body-selective regions

Area of Science:

• Neuroscience
• Computer Vision
• Cognitive Science

Background:

• Neural networks trained on natural language encode complex scene semantics.
• Interpreting these semantic representations in the brain remains challenging.

Purpose of the Study:

• To investigate the relationship between object co-occurrence in natural scene captions and brain activity.
• To characterize the aspects of natural scenes contributing to semantic representations in category-selective brain regions.

Main Methods:

• Developed an analysis framework linking object co-occurrence in scene captions with fMRI responses.
• Utilized caption-based encoding models to predict brain activity.
• Focused analysis on the extrastriate body area (EBA) and fusiform body area.

Main Results:

• Human bodies co-occurring with sports objects elicited the strongest predicted responses in the EBA.
• Bodies co-occurring with vehicles or accessories showed strong but weaker responses.
• Identified motion speed, number of people, and body size as key contributors to EBA semantic representation.

Conclusions:

• Object co-occurrence analysis provides an interpretable method for understanding visual representations.
• The framework helps elucidate how the brain processes semantic information in natural scenes.
• Body motion speed is a primary feature driving semantic representation in body-selective areas.