Shared representations of human actions across vision and language

Area of Science:

• Cognitive Science
• Neuroscience
• Computational Linguistics

Background:

• Humans excel at recognizing and communicating actions.
• Understanding how actions are mentally organized across different modalities (vision, language) is crucial.

Purpose of the Study:

• To investigate the mental organization of human actions.
• To determine if this organization is shared across visual and linguistic representations.
• To explore the role of action targets and semantic features in this organization.

Main Methods:

• Collected similarity judgments for human actions from naturalistic videos and sentences.
• Tested four computational models of action categorization at varying abstraction levels.
• Utilized language model embeddings to predict behavioral similarities.

Main Results:

• A shared organization of action representations was found across video and sentence modalities.
• Action target (object, person, self) was the primary determinant of this shared organization.
• Language model embeddings successfully predicted behavioral similarities and captured action target information.

Conclusions:

• Action concepts are organized similarly in the mind for both vision and language.
• This shared organization is mainly driven by socially relevant goals, particularly the target of actions.
• Language models can effectively represent and predict human action understanding across modalities.