Development of compositionality through interactive learning of language and action of robots

Area of Science:

• Robotics
• Computational Neuroscience
• Artificial Intelligence

Background:

• Human generalization relies on compositionality, breaking down tasks into reusable parts.
• Robotics faces challenges in developing linguistic compositionality alongside sensorimotor skills via associative learning, especially with partial data.

Purpose of the Study:

• Propose a brain-inspired neural network model for integrated sensorimotor and linguistic learning.
• Investigate how associative learning and partial data impact compositional skill development in robots.

Main Methods:

• Developed a neural network integrating vision, proprioception, and language using predictive coding and active inference (free-energy principle).
• Assessed model effectiveness through simulations with a robot arm performing various tasks.
• Conducted ablation studies to identify critical components like visual attention and working memory.

Main Results:

• Increased training variations in task composition significantly enhanced generalization to unlearned verb-noun commands.
• Self-organized compositional structures in the linguistic latent state space were substantially influenced by sensorimotor learning.
• Visual attention and working memory were found essential for accurate visuomotor sequence generation to meet linguistic goals.

Conclusions:

• The proposed model demonstrates enhanced generalization in robots through integrated linguistic and sensorimotor learning.
• Sensorimotor experience plays a crucial role in shaping compositional structures within the linguistic latent space.
• Findings provide insights into the developmental mechanisms of compositionality through multimodal interactions.