Natural language instructions induce compositional generalization in networks of neurons

Area of Science:

• Cognitive Science
• Neuroscience
• Artificial Intelligence

Background:

• Human ability to perform novel tasks based on linguistic instructions is a fundamental cognitive feat.
• Neural mechanisms underlying this generalization capability remain poorly understood.
• Advances in natural language processing offer new avenues for modeling cognitive functions.

Purpose of the Study:

• To develop and evaluate a neural model capable of performing novel tasks solely based on linguistic instructions.
• To investigate how language scaffolds sensorimotor representations for flexible task execution.
• To explore the potential for AI models to generate task descriptions from motor feedback.

Main Methods:

• Utilized advances in natural language processing and pre-trained language models.
• Trained neural network models on a set of common psychophysical tasks with embedded linguistic instructions.
• Evaluated model performance on unseen tasks using a zero-shot learning paradigm.

Main Results:

• The best models achieved an average performance of 83% correct on novel tasks using only linguistic instructions (zero-shot learning).
• Demonstrated that language scaffolds sensorimotor representations, aligning task activity geometry with instruction semantics.
• Showcased a model's ability to generate linguistic descriptions of novel tasks from motor feedback.

Conclusions:

• Language plays a crucial role in scaffolding sensorimotor representations for generalized cognitive abilities.
• Neural models can effectively learn to perform novel tasks from linguistic instructions, mimicking human generalization.
• The findings provide experimentally testable predictions for understanding the neural basis of language-guided flexible cognition.