The HA4M dataset: Multi-Modal Monitoring of an assembly task for Human Action recognition in Manufacturing

Area of Science:

• Computer Vision
• Machine Learning
• Robotics

Background:

• Assembly tasks in manufacturing require sophisticated human action recognition.
• Existing datasets may lack the multi-modal data richness needed for complex assembly analysis.

Purpose of the Study:

• Introduce the Human Action Multi-Modal Monitoring in Manufacturing (HA4M) dataset.
• Provide a comprehensive multi-modal dataset for human action recognition in assembly tasks.

Main Methods:

• Collected multi-modal data from 41 subjects performing an Epicyclic Gear Train (EGT) assembly task.
• Utilized a Microsoft Azure Kinect for capturing RGB, Depth, and InfraRed (IR) data.
• Generated aligned RGB-to-Depth images, Point Clouds, and Skeleton data.

Main Results:

• The HA4M dataset is the first multi-modal dataset specifically for an assembly task.
• It comprises six types of data: RGB, Depth, IR, aligned RGB-to-Depth, Point Clouds, and Skeleton data.
• The dataset includes 41 subjects performing 12 distinct actions across multiple trials.

Conclusions:

• The HA4M dataset is a valuable resource for developing and testing advanced action recognition systems.
• Facilitates research in smart manufacturing and human-robot collaboration.
• Enables deeper insights into human actions during complex assembly processes.