Incremental learning of humanoid robot behavior from natural interaction and large language models.

Area of Science:

• Robotics
• Artificial Intelligence
• Human-Robot Interaction

Background:

• Natural language dialog is crucial for intuitive human-robot interaction, allowing for intent expression and error correction.
• Robots need to learn incrementally from interaction experiences to improve performance and avoid future mistakes.

Purpose of the Study:

• To propose and demonstrate a system for incremental learning of complex, high-level robot behavior from natural language interactions.
• To enable robots to learn from their mistakes and improve behavior through continuous feedback loops.

Main Methods:

• Utilizing large language models (LLMs) for high-level behavior orchestration, generating Python statements to control robot perception and action.
• Implementing an incremental learning mechanism where LLMs learn from human instructions, environmental observations, and execution results.
• Employing a secondary LLM for code-level improvements based on human feedback to correct misunderstandings.

Main Results:

• Demonstrated successful implementation on a humanoid robot (ARMAR-6), showcasing generalized incrementally learned knowledge.
• Validated the system's effectiveness through quantitative (simulation) and qualitative (simulation and real-world) evaluations.
• Showcased the ability to store and retrieve improved interactions for semantically similar future requests.

Conclusions:

• The proposed system effectively enables robots to learn complex behaviors incrementally from natural language interactions.
• LLM-powered feedback loops and error correction mechanisms significantly enhance robot adaptability and learning capabilities.
• This approach advances human-robot collaboration by allowing robots to continuously refine their understanding and actions based on user input.