Action Recognition by an Attention-Aware Temporal Weighted Convolutional Neural Network

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Human action recognition research has advanced with Convolutional Neural Networks (CNNs).
• Integrating temporal information into CNNs for video analysis remains an active research area.
• Recurrent attention models from natural language processing inspire new approaches.

Purpose of the Study:

• To propose an effective and efficient method for incorporating temporal dynamics into CNNs for action recognition.
• To introduce the Attention-aware Temporal Weighted CNN (ATW CNN) framework.
• To enhance video representation by focusing on salient temporal segments.

Main Methods:

• Developed the Attention-aware Temporal Weighted CNN (ATW CNN) by embedding a visual attention model into a temporal weighted multi-stream CNN.
• Implemented the attention mechanism as temporal weighting.
• Utilized stochastic gradient descent (SGD) with back-propagation for end-to-end training of network parameters and temporal weights.

Main Results:

• The proposed attention mechanism significantly boosts recognition performance.
• The ATW CNN effectively focuses on more relevant video segments, yielding more discriminative representations.
• Experimental validation on UCF-101 and HMDB-51 datasets demonstrates substantial performance gains.

Conclusions:

• The ATW CNN provides a powerful approach for human action recognition in videos.
• Temporal weighting via an attention mechanism is a simple yet effective strategy for improving CNN-based video analysis.
• The method enhances the discriminative power of video representations by prioritizing relevant temporal information.