Enhancing Human-Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object...

Area of Science:

• Robotics
• Human-Computer Interaction
• Artificial Intelligence

Background:

• Robots are increasingly integrated into daily life, necessitating intuitive interaction interfaces.
• Effective human-robot collaboration enhances user experience and task reliability.
• Current interfaces often lack the naturalness and multi-modal understanding required for complex tasks.

Purpose of the Study:

• To propose a multi-modal framework for natural and intuitive human-robot collaborative interaction.
• To enhance user experience and task reliability in human-robot collaboration.
• To integrate various sensors and AI modules for seamless robotic task execution.

Main Methods:

• A multi-modal framework with independent modules utilizing image, sound, and depth sensors.
• Object location and pose determination using feature-detector-descriptor and homography-based techniques.
• Extraction of verbal instructions via deep multi-task learning, user of interest (UOI) detection, and gesture/gaze estimation.

Main Results:

• The framework effectively integrates object recognition, verbal instruction interpretation, UOI identification, and gesture/gaze analysis.
• Demonstrated the capability to form structured commands for robots by combining multi-modal inputs.
• Experimental results confirmed the effectiveness of the proposed interaction interfaces.

Conclusions:

• The developed multi-modal framework significantly improves human-robot collaborative interaction.
• The system facilitates natural and intuitive communication, leading to enhanced task reliability.
• This approach offers a robust solution for designing advanced human-robot interaction systems.