Convolutional Block Attention Module-Multimodal Feature-Fusion Action Recognition: Enabling Miner Unsafe Action...

Area of Science:

• Computer Vision
• Artificial Intelligence
• Mining Safety Engineering

Background:

• Unsafe miner actions are a primary cause of mine accidents.
• Real-time recognition of these actions is crucial for accident prevention.
• Existing computer vision methods require enhancement for accurate detection in mining environments.

Purpose of the Study:

• To develop an advanced computer vision model for accurate and real-time recognition of unsafe actions performed by underground miners.
• To improve the performance of existing action recognition models by incorporating attention mechanisms and multimodal fusion.

Main Methods:

• Construction of the Unsafe Actions of Underground Miners (UAUM) dataset with ten action categories.
• Image enhancement using spatial- and frequency-domain algorithms.
• Integration of YOLOX object detection and Lite-HRNet key-point detection for skeleton data extraction.
• Development of the Convolutional Block Attention Module-Multimodal Feature-Fusion Action Recognition (CBAM-MFFAR) model, combining skeleton (CBAM-PoseC3D) and RGB (CBAM-SlowOnly) modalities.

Main Results:

• The CBAM-MFFAR model achieved 95.8% accuracy on the NTU60 RGB+D dataset and 94.6% on the UAUM dataset.
• Demonstrated significant accuracy improvements over existing models like CBAM-PoseC3D, PoseC3D, 2S-AGCN, and ST-GCN.
• Successfully validated in field trials, accurately recognizing complex and multiple unsafe actions in real mining sites.

Conclusions:

• The CBAM-MFFAR model offers a robust and accurate solution for real-time unsafe action recognition in underground mining.
• The multimodal approach and attention mechanism significantly enhance recognition performance.
• This technology has the potential to substantially improve safety protocols and reduce accidents in the mining industry.