Towards Recognition of Human Actions in Collaborative Tasks with Robots: Extending Action Recognition with Tool...

Area of Science:

• Robotics and Automation
• Computer Vision
• Human-Computer Interaction

Background:

• Manual assembly processes require efficient tool recognition for progress tracking and human action recognition (HAR) in collaborative tasks.
• Existing methods for progress detection and tool recognition have limitations in real-time, integrated applications.

Purpose of the Study:

• To develop and implement a novel online tool recognition method for handheld tools in manual assembly.
• To integrate this method with existing Human Action Recognition (HAR) systems for collaborative tasks.
• To evaluate the generalizability and performance of the proposed pipeline.

Main Methods:

• A two-stage pipeline was developed: 1. Region of Interest (ROI) extraction using wrist position from skeletal data. 2. Tool classification within the cropped ROI.
• Utilized various object recognition algorithms and image classification approaches.
• Created an extensive training dataset for tool recognition.

Main Results:

• The pipeline demonstrated generalizability across different algorithms and tool classes.
• Offline evaluation included twelve tool classes, showing competitive prediction accuracy and robustness.
• Online tests in assembly scenarios, with varied instances and backgrounds, confirmed the system's capability.

Conclusions:

• The novel online tool recognition pipeline is effective and competitive with existing approaches.
• The method offers advantages in accuracy, robustness, diversity, and online capability for manual assembly.
• This work advances the integration of vision-based tool recognition into HAR systems for collaborative robotics.