Foundation models assist in human-robot collaboration assembly.

Area of Science:

• Robotics
• Artificial Intelligence
• Manufacturing Technology

Background:

• Existing human-robot collaboration (HRC) systems struggle with environmental perception and task reasoning, requiring specialized models and retraining for new objects or tasks.
• Current HRC methods are limited to predefined tasks and cannot handle undefined reasoning, hindering flexibility and generalization.

Purpose of the Study:

• To propose a novel HRC approach leveraging Foundation Models (FMs), specifically Large Language Models (LLMs) and Vision Foundation Models (VFMs).
• To overcome the limitations of current HRC systems in environment perception and task reasoning for enhanced flexibility and generalization.

Main Methods:

• Utilized LLMs with prompt learning for undefined task reasoning in HRC.
• Integrated various VFMs for scene semantic perception without the need for model retraining.
• Developed a comprehensive FM-based HRC system incorporating perception, reasoning, and execution modules.

Main Results:

• Demonstrated superior performance of FMs in perception and reasoning through extensive experiments.
• Validated the feasibility and effectiveness of the FM-based HRC system in a part assembly case study.
• Achieved more flexible and generalized HRC capabilities compared to existing methods.

Conclusions:

• Foundation Models offer a powerful solution to enhance HRC systems' perception and reasoning capabilities.
• The proposed FM-based HRC system provides a flexible and generalized approach for customized manufacturing tasks.
• This work paves the way for more adaptable and intelligent human-robot collaboration in manufacturing.