Large language models without grounding recover non-sensorimotor but not sensorimotor features of human concepts

Area of Science:

• Cognitive Science
• Artificial Intelligence
• Computational Linguistics

Background:

• Large language models (LLMs) are increasingly sophisticated, prompting questions about their ability to form complex conceptual representations.
• The necessity of grounding in multisensory experience for concept formation remains a key debate.
• LLMs without grounding challenge traditional views on the role of embodied experience in cognition.

Purpose of the Study:

• To investigate the extent to which LLMs can generate complex conceptual representations.
• To compare conceptual representations between humans and state-of-the-art LLMs.
• To determine the impact of visual learning on LLM conceptual representation alignment with humans.

Main Methods:

• Compared multidimensional representations of ~4,442 lexical concepts.
• Utilized human datasets (Glasgow Norms, Lancaster Norms) and LLMs with/without visual learning.
• Analyzed representations across non-sensorimotor, sensory, and motor domains.

Main Results:

• Similarity between LLM and human representations decreased from non-sensorimotor to sensory and motor domains, indicating divergence.
• LLMs with visual learning showed enhanced similarity with human representations in visual-related dimensions.
• A systematic divergence was observed, particularly in motor domains.

Conclusions:

• Language alone may have limitations for LLMs in achieving human-like conceptual representation.
• Integrating diverse modalities, such as visual learning, can enhance LLM alignment with human conceptual systems.
• Multisensory experience may be crucial for developing robust, human-like conceptual understanding in AI.