Embodiment in multimodal large language models

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Robotics

Background:

• Multimodal large language models (MLLMs) excel at integrating text and visual data.
• Current MLLMs struggle with situated interactions in complex, real-world environments.
• Embodied experience is crucial for understanding and interacting within these environments.

Purpose of the Study:

• To propose a framework for enhancing MLLMs with internal and external embodiment.
• To address the limitations of current MLLMs in real-world situated interactions.
• To bridge the gap between multimodal data processing and genuine world experience.

Main Methods:

• Describing internal and external embodiment mechanisms in humans.
• Relating human embodiment to current MLLM advancements.
• Proposing a dual-embodied framework for MLLM development.

Main Results:

• Identified the necessity of internal and external embodiment for advanced MLLMs.
• Highlighted the early stages of MLLM alignment with human representations.
• Introduced a dual-embodied framework to model embodiment interactions.

Conclusions:

• Incorporating internal and external embodiment is key for future MLLM development.
• A dual-embodied framework can enhance MLLMs' ability to bridge data and world experience.
• This approach moves MLLMs closer to human-like situated interaction capabilities.