Ensemble Encoder-Enabled Proactive Human Assembly Intention Recognition With Multimodal and Flexible Scale Data.

Area of Science:

• Robotics
• Artificial Intelligence
• Computer Vision

Background:

• Human-robot collaboration (HRC) assembly requires accurate mutual understanding for safety and efficiency.
• Human assembly intention recognition (HAIR) is crucial for HRC, but current methods struggle with limited industrial data, varying scales, and occlusions.

Purpose of the Study:

• To propose an ensemble encoder approach for improved HAIR in HRC assembly.
• To effectively extract and fuse spatiotemporal features from visual and skeleton data under complex conditions.

Main Methods:

• Developed an RGB feature extraction encoder (RGBE) with cross-attention for multiscale feature fusion.
• Designed a mask-aware skeleton feature extraction encoder to handle occlusions using frame and joint masking.
• Integrated features using a global feature fusion encoder for comprehensive action representation.

Main Results:

• Achieved state-of-the-art accuracy: 99.12% on MCV-Intention, 99.23% on HA4M, and 84.59% on HA-VID.
• Demonstrated robust performance under occlusion and varying illumination conditions.
• Ablation studies confirmed the effectiveness of fusion strategies and components.

Conclusions:

• The proposed ensemble encoder significantly enhances HAIR accuracy and efficiency in HRC assembly.
• The method effectively addresses challenges like limited data, varying scales, and visual occlusions.
• This approach advances the development of more intelligent and reliable collaborative robotic systems.