Cross-category spatiotemporal consensus and discriminative networks for weakly-supervised temporal action...

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Weakly-supervised temporal action localization (WS-TAL) aims to identify action instances in untrimmed videos without frame-level annotations.
• Existing WS-TAL methods often focus on enhancing action snippet features or exploring non-salient regions, but neglect cross-category consensus and higher-order dynamics.
• These overlooked aspects, cross-category consensus relationships and class-level higher-order dynamics, offer crucial information for comprehensive localization and capturing subtle discriminative features.

Purpose of the Study:

• To develop a novel method, the STCD network, for improved weakly-supervised temporal action localization.
• To leverage superclass-level semantics and high-order dynamics for spatiotemporal consensus and discriminative learning.
• To enhance the discriminative features of action snippets by reducing uncertainty and minimizing information entropy.

Main Methods:

• The STCD network utilizes a high-order encoding module based on Koopman theory to explore discriminative class-wise dynamics.
• Superclass-level semantics are employed to capture consensus relationships among actions with similar sub-actions.
• An information-theoretic loss function is proposed to reduce the uncertainty of ambiguous action snippets by minimizing their information entropy.

Main Results:

• The proposed STCD network demonstrates superior performance compared to state-of-the-art methods on benchmark datasets.
• Experimental results on THUMOS14, ActivityNet v1.2, and ActivityNet v1.3 validate the effectiveness of the approach.
• The method successfully leverages cross-category consensus and higher-order dynamics for more comprehensive action localization.

Conclusions:

• The STCD network offers a simple yet effective approach for weakly-supervised temporal action localization.
• Incorporating superclass semantics and high-order dynamics significantly enhances localization accuracy.
• The information-theoretic loss function effectively improves the discriminative power of action features.