Confidence-Aware Gated Multimodal Fusion for Robust Temporal Action Localization in Occluded Environments

Area of Science:

• Computer Vision
• Machine Learning
• Robotics

Background:

• Robust Temporal Action Localization (TAL) is critical for industrial monitoring.
• Skeletal data in industrial settings suffers from occlusions, hindering multimodal fusion.
• Existing methods often fail due to unreliable skeletal data transfer.

Purpose of the Study:

• To develop a novel Gated Skeleton Refinement Module (Gated SRM) for robust TAL.
• To address the negative transfer issue caused by occluded skeletal data.
• To improve the reliability of multimodal fusion in challenging industrial environments.

Main Methods:

• Proposed a Gated Skeleton Refinement Module (Gated SRM) as a front-end preprocessing step.
• Incorporated OpenPose confidence scores as a logarithmic bias in a self-attention mechanism for soft suppression of unreliable joints.
• Utilized a learnable gating network for adaptive fusion of refined skeletal features with RGB data.

Main Results:

• Achieved a mean Average Precision (mAP) of 21.77% on the IKEA ASM dataset, demonstrating superior robustness under occlusion.
• Effectively prevented accuracy degradation common in naive and established multimodal fusion strategies.
• Maintained an end-to-end inference speed of 9.2 FPS, suitable for industrial workflow monitoring.

Conclusions:

• The Gated SRM offers a robust and principled sensor-metadata-driven approach for action recognition under occlusion.
• Prioritizing confidence-based data selection over data restoration provides a critical fail-safe for real-world applications.
• The method enhances multimodal fusion reliability without compromising practical inference speeds.