Enhancing robotic skill acquisition with multimodal sensory data: A novel dataset for kitchen tasks

Area of Science:

• Robotics
• Human-Computer Interaction
• Artificial Intelligence

Background:

• Large language models (LLMs) in human-robot interaction (HRI) are limited by unimodal data.
• Integrating environmental, physiological, and physical data is crucial for advanced HRI.
• Current datasets lack the complexity needed for real-world embodied AI.

Purpose of the Study:

• To develop novel multimodal data collection methodologies for HRI.
• To capture the complexity of human interaction in kitchen environments.
• To create a comprehensive dataset for embodied AI and skill learning in robotics.

Main Methods:

• Collected data from 20 adults using 17 kitchen tools in dynamic scenarios.
• Acquired multimodal data including tactile, EMG, audio, whole-body movement, and eye-tracking.
• Generated a dataset of 680 segments (~11 hours) with 56,000 annotations across seven modalities.

Main Results:

• Successfully created a rich, multimodal dataset of human-robot interaction.
• The dataset captures diverse human sensory and motion data during task execution.
• Provides a benchmark for utility and repeatability in robotic skill learning.

Conclusions:

• This work bridges the gap between real-world multimodal data and embodied AI.
• The dataset facilitates the development of more capable and adaptable robots.
• Establishes a new standard for skill learning research in robotics.