Visual and Affective Multimodal Models of Word Meaning in Language and Mind.

Area of Science:

• Cognitive Science
• Computational Linguistics
• Psycholinguistics

Background:

• Natural language processing (NLP) models often lack multimodal representations, unlike human cognition.
• Current models struggle to fully capture the nuances of both concrete and abstract concept representation.
• Existing linguistic models are primarily unimodal, relying solely on text or word association data.

Purpose of the Study:

• To evaluate how well unimodal and multimodal models capture human representations of concepts.
• To compare linguistic models (text corpora vs. word associations) with multimodal models (linguistic + perceptual/affective).
• To investigate the impact of visual and affective information on semantic representation models.

Main Methods:

• Conducted two new studies and reanalyzed previous research.
• Compared unimodal distributional linguistic models with multimodal models integrating perceptual or affective data.
• Analyzed models based on text corpora versus those derived from word association data.

Main Results:

• Multimodal models (visual and affective) significantly outperformed unimodal linguistic models in capturing human concept representations.
• The advantage of multimodal models was particularly strong for basic-level concepts within the same superordinate category.
• Integrating visual and affective features enhanced text-based linguistic models more than word association-based models.

Conclusions:

• Multimodal information, including visual and affective data, is crucial for accurately representing both abstract and concrete word meanings.
• Purely linguistic information is insufficient for a complete understanding of word meaning, necessitating multimodal integration.
• Findings suggest qualitative differences in information encoded by natural language corpora versus word associations.