Everything robots need to know about cooking actions: creating actionable knowledge graphs to support robotic meal...

Area of Science:

• Robotics
• Artificial Intelligence
• Natural Language Processing

Background:

• Robots currently lack the ability to autonomously prepare meals from natural language instructions.
• Bridging the gap between human-understandable recipes and robotic execution is a significant challenge.

Purpose of the Study:

• To develop a novel methodology for enabling robots to autonomously prepare meals by interpreting natural language recipe instructions.
• To map recipe instructions to robotic manipulation tasks using Actionable Knowledge Graphs (AKGs).

Main Methods:

• Leveraging Actionable Knowledge Graphs (AKGs) to categorize recipe instructions into six core robotic manipulation tasks (Action Cores).
• Utilizing the Recipe1M+ dataset to analyze action verbs and map them to Action Cores via direct matching and cosine similarity.
• Employing a neuro-symbolic approach with Large Language Models to handle unmatched verbs and generate new action cores.

Main Results:

• Achieved 76.5% coverage in matching recipe action verbs to defined Action Cores.
• Demonstrated the versatility of AKGs in adapting general plans to specific robotic tasks.
• Successfully validated the methodology through an experimental meal preparation scenario.

Conclusions:

• Actionable Knowledge Graphs provide a versatile framework for translating recipes into robotic actions.
• This approach lays the foundation for adaptive robotic systems capable of complex culinary tasks with minimal human intervention.
• The neuro-symbolic method enhances the system's ability to interpret and execute a wider range of recipe instructions.