Structured Attention Composition for Temporal Action Localization.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Temporal action localization in untrimmed videos is crucial for understanding video content.
• Existing methods often treat appearance and motion features equally, leading to suboptimal performance.
• Different actions have varying dependencies on appearance versus motion cues.

Purpose of the Study:

• To propose a novel multi-modality feature learning approach for temporal action localization.
• To develop a structured attention composition module that adaptively weights appearance and motion features.
• To improve the precision of localizing action instances in untrimmed videos.

Main Methods:

• A novel structured attention composition module is introduced, integrating frame and modality attention.
• This module learns a frame-modality structure using optimal transport theory.
• It regularizes individual frame and modality attention inferences for better feature representation.

Main Results:

• The proposed module consistently improves existing state-of-the-art temporal action localization methods.
• New state-of-the-art performance is achieved on the THUMOS14 benchmark.
• Experiments demonstrate the effectiveness of adaptive feature weighting for action localization.

Conclusions:

• The structured attention composition module offers a plug-and-play solution for enhancing temporal action localization.
• Adaptive weighting of appearance and motion features is key to improving model performance.
• This approach advances the field by better exploiting multi-modality feature learning.