Affordance embeddings for situated language understanding.

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Computational Linguistics

Background:

• Deep learning models in AI have advanced NLP but lack transferability to novel situations.
• Current AI systems often fail to generalize beyond their training data, limiting real-world applicability.

Purpose of the Study:

• To address AI's transfer learning limitations by computationally grounding linguistic information.
• To develop a neurosymbolic approach for creating flexible AI models capable of situated understanding.

Main Methods:

• Utilizing a neurosymbolic perspective with multimodal contextual modeling of interactive situations.
• Combining neural and symbolic methods with multimodal simulations in the VoxWorld platform.
• Encoding object affordances and habitats within a computational framework.

Main Results:

• Demonstrated that neural embedding vectors of symbolically-encoded object affordances facilitate knowledge transfer.
• Showcased the ability to transfer knowledge of objects and situations to novel entities.
• Enabled learning for recognizing and generating linguistic and gestural denotations.

Conclusions:

• Computational situated grounding offers a solution to AI's transfer learning challenges.
• Neurosymbolic AI combined with multimodal simulations can create more flexible and adaptable AI systems.
• The VoxWorld platform provides a viable method for modeling context-aware communication in AI.